There are two methods for importing data into Slate via a Web Service, depending on how data is made available by the third-party source or middleware product:

1. Push data into Slate's web service endpoint

2. Pull data from the third party/middleware's web service endpoint into Slate

Slate has no preference on whether you choose to push or pull data. Make the choice based on the tools offered by the third-party data source or middleware product.

Creating a source format

Regardless of your decision, you will need to construct a source format to consume the data into Slate. The initial set-up and mapping of data points are the same no matter the import mechanism (web service push, pull, or SFTP file transfer).

To configure a source format, see Creating a New Custom Source Format.

Once you have created and tested your source format to process the data from the web service, you can configure the format to automatically push or pull data as desired using the instructions that follow.

📝 Note

• Upload Dataset works best with batch processes. If your third-party data source provides individual row updates, look into batching records together. This may require a middleware solution.

• The Upload Dataset rate limit is 50 posts in a 5 minute window.

• Most uploads are processed within 15 minutes, so while you should see the source within Upload Dataset immediately, you may not see the imported data for a few minutes.

Pulling data from a web service into Slate

1. Under Database, search for Source Formats and open your web service source format.

2. Edit the format and navigate to the Import Automation tab.

3. In the Import Remote Server field, enter the URL of the web service to be called. The third-party source will provide the URL to access the web service. There are, usually, basic authentication credentials to access the remote server which must be included in the URL by placing the username, colon, password, and an @ symbol before the URL (e.g., https://username:password@studentdata.vendor.com). Optionally, the third-party source may require you to send an authorization or other headers to access your data. See the next step for additional details.

4. If required, use the HTTP Headers setting to add authorization and other required headers as defined by the third party.

5. (Optional) By default, Slate will pull data from the web service every 24 hours. If your use case requires more frequent pulls, you can use the Import Frequency (mins) setting to increase the frequency by specifying the desired number of minutes between pulls. N

   📝 Note

   Upload Dataset operates most efficiently as a batch import. The minimum interval between pulls is 60 minutes. Slate will pull data every 60 minutes for any value less than that.

6. Click Save.

Pushing data from a web service into Slate

To push data into Slate, the third-party web service can post the data to a special web service endpoint that is unique to the source format that will process the data. There are two options depending on whether you wish to use Basic Authentication (standard) or Certificate Authentication.

In either case, you'll need a Security Administrator to configure a Service Account user account for authentication. In addition to the routine steps to configure a User Account, the Security Administrator should select a User Type of Service Account and either add a Password or Client Certificate, depending on the desired authentication method.

Once the Service Account user has been configured, you may securely share the username, authentication details, and URL to the web service source format with the third party. Use the following steps to find the URL.

1. Under Database, search for Source Formats and open your web service source format.

2. In the overview of the format, click the View link next to Web Services.

3. For Basic Authentication, use the URL found on the Standard tab. When making the POST request to this URL, the third party must include an Authorization header which is the Base64 encoding of "username:password".

   ✨ Tip

   Add the query string parameter &filename=EXAMPLE_FILE to the URL to customize the incoming file name.

4. For Certificate Authentication, use the URL found on the Certificate-Based Authentication tab. When making the POST request to this URL, the third party must include an Authorization header which is the Base64 encoding of "username" as well as an X-Client-Certificate header with the certificate.

Establishing Source Import Callbacks using a WebHook URL

This option lets Slate send a Web Service HTTP POST to an external system. This POST includes a JSON Payload and confirms that the import was successful in Slate. The external system receiving this confirmation can use it to mark its records as exported and updated in Slate.

To add a web callback URL to a source format:

1. From the main navigation, select Database.

2. From the section Import and Data Processing, select Sources / Upload Dataset.

3. On the right, click Source Formats.

4. Select the source format to which you'd like to add a WebHook URL.

5. On the right, click Edit.

6. Select the tab Import Automation. 

7. Paste your Web Callback URL into the Web Callback URL field.

8. Click Save.

   

Below is an example of a POST. For any records not updated or exported, a process can be implemented within your institution that prompts a member of your team to review and determine the root cause.

  {
    "status": "success",
    "source": "d13a8c60-d6da-2844-4704-0d3d78eed25f",
    "summary": "Common App Prospects",
    "database": "XYZ",
    "started" : "2022-06-28T06:12:22.345Z",
    "stopped" : "2022-06-28T06:12:23.577Z"
   }