There are two methods for importing data into Slate via a Web Service, depending on how data is made available by the third-party source or middleware product:

• Pull data from the third party or middleware's web service endpoint into Slate

• Push data into Slate's web service endpoint

Slate has no preference on whether you pull or push data. Make the choice based on the tools offered by the third-party data source or middleware product.

Creating a source format

Regardless of your decision, first construct a source format to consume the data into Slate.

The initial set-up and mapping of data points are the same regardless of import mechanism, whether it be web service push, pull, or SFTP file transfer.

To configure a source format, see Creating a New Custom Source Format.

Once you have created and tested your source format to process the data from the web service, you can configure the format to automatically push or pull data as desired using the instructions that follow.

Notes on Upload Dataset

• Upload Dataset works best with batch processes. If your third-party data source provides individual row updates, look into batching records together. This may require a middleware solution.

• The Upload Dataset rate limit is 50 posts in a 5 minute window.

• Most uploads are processed within 15 minutes, so while you should see the source within Upload Dataset immediately, you may not see the imported data for a few minutes.

Option 1: Pulling data from a web service into Slate

1. From the main navigation, select Database.

2. Select Source Formats.

3. Select the web service source format you created in Creating a source format.

4. Select Edit.

5. Select the Import Automation tab.

6. Configure the following settings:

   • Import Remote Server: Enter the URL of the web service to be called.

     • The third-party source provides the URL to access the web service.

     • There are, usually, basic authentication credentials to access the remote server which must be included in the URL by placing the username, colon, password, and an @ symbol before the URL. For example, https://username:[email protected].

   • HTTP Headers: The third-party source may require you to send an authorization or other headers to access your data. If required, use the HTTP Headers setting to add authorization and other required headers as defined by the third party.

   • (Optional) Import Frequency (mins): By default, Slate pulls data from the web service every 24 hours. If your use case requires more frequent pulls, enter an Import Frequency in minutes to increase the frequency to the desired number of minutes between pulls.

     📝 Note

     Upload Dataset operates most efficiently as a batch import. The minimum interval between pulls is 60 minutes. Slate will pull data every 60 minutes for any value less than that.

   • (Optional) Web Callback URL: This setting lets Slate send a Web Service HTTP POST to an external system.

     • This POST includes a JSON Payload and confirms that the import was successful in Slate.

     • The external system receiving this confirmation can use it to mark its records as exported and updated in Slate.

     • The following JSON code is an example of a POST. For any records not updated or exported, a process can be implemented within your institution that prompts a member of your team to review and determine the root cause.

         {
           "status": "success",
           "source": "d13a8c60-d6da-2844-4704-0d3d78eed25f",
           "summary": "Common App Prospects",
           "database": "XYZ",
           "started" : "2022-06-28T06:12:22.345Z",
           "stopped" : "2022-06-28T06:12:23.577Z"
          }
       

       sample_xml_import_file.xml

7. Select Save.

Option 2: Pushing data from a web service into Slate

To push data into Slate, the third-party web service can post the data to a special web service endpoint that is unique to the source format that will process the data. There are two options depending on whether you wish to use Basic Authentication (standard) or Certificate Authentication.

In either case, you'll need a Security Administrator to configure a Service Account user account for authentication. In addition to the routine steps to configure a User Account, the Security Administrator should select a User Type of Service Account and either add a Password or Client Certificate, depending on the desired authentication method.

Once the Service Account user has been configured, you may securely share the username, authentication details, and URL to the web service source format with the third party. Use the following steps to find the URL.

1. From the main navigation, select Database.

2. Select Source Formats.

3. Select the web service source format you created in Creating a source format.

4. Next to Web Services, select View.

5. Depending on your use case:

   • If using Basic Authentication:

     • Copy the URL found on the Standard tab.

     • When making the POST request to this URL, the third party must include an Authorization header which is the Base64 encoding of username:password.

       ✨ Tip

       Add the query string parameter &filename=EXAMPLE_FILE to the URL to customize the incoming file name.

   • If using Certificate Authentication:

     • Copy the URL found on the Certificate-Based Authentication tab.

     • When making the POST request to this URL, the third party must include an Authorization header which is the Base64 encoding of username.

6. Select Close.

Code Samples

The following code samples provide examples of JSON file posts using different languages and authentication methods.

Basic Authentication

These code samples post a sample JSON file using basic authentication and the credentials username:password.

const axios = require('axios');
let data = JSON.stringify({
  "row": {
    "fieldKey": "fieldData"
  }
});

let config = {
  method: 'post',
  maxBodyLength: Infinity,
  url: 'https://slate-university.test.technolutions.net/manage/service/import?cmd=load&format=11c409f5-7bc5-4cf2-bbb9-1264af6d39ee',
  headers: { 
    'Authorization': 'Basic dXNlcm5hbWU6cGFzc3dvcmQ=', 
    'Content-Type': 'application/json'
  },
  data : data
};

axios.request(config)
.then((response) => {
  console.log(JSON.stringify(response.data));
})
.catch((error) => {
  console.log(error);
});

var client = new HttpClient();
// use basic authentication url
var request = new HttpRequestMessage(HttpMethod.Post, "https://slate-university.test.technolutions.net/manage/service/import?cmd=load&format=11c409f5-7bc5-4cf2-bbb9-1264af6d39ee");
//base64 encoding of "username:password"
request.Headers.Add("Authorization", "Basic dXNlcm5hbWU6cGFzc3dvcmQ=");
// build a JSON payload
var content = new StringContent("{\n    \"row\": \n    { \n        \"fieldKey\": \"fieldData\"\n    }\n}", null, "application/json");
request.Content = content;
var response = await client.SendAsync(request);
response.EnsureSuccessStatusCode();
Console.WriteLine(await response.Content.ReadAsStringAsync());

import requests
import json

# use the basic authentication url
url = "https://slate-university.test.technolutions.net/manage/service/import?cmd=load&format=11c409f5-7bc5-4cf2-bbb9-1264af6d39ee"

#build a JSON payload
payload = json.dumps({
  "row": {
    "fieldKey": "fieldData"
  }
})

# add headers
headers = {
  'Authorization': 'Basic dXNlcm5hbWU6cGFzc3dvcmQ=',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(str(response.status_code) + ": " + response.text)

curl --location 'https://slate-university.test.technolutions.net/manage/service/import?cmd=load&format=11c409f5-7bc5-4cf2-bbb9-1264af6d39ee' \
--header 'Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=' \
--header 'Content-Type: application/json' \
--data '{
    "row": 
    { 
        "fieldKey": "fieldData"
    }
}'
echo "\nthis one is for greybeards who like Unix"

Certificate Authentication

These code samples post a sample JSON file using certificate authentication.

📝 Note: The C# example uses (and requires) a single .pfx file to represent the certificate and key. The remaining examples use separate .pem files.

const axios = require('axios');
const fs = require('fs');
const https = require('https')

// use certificate authentication URL
const url = 'https://cluster-mtls.technolutions.net/slate-university.test.technolutions.net/manage/service/import?cmd=load&format=11c409f5-7bc5-4cf2-bbb9-1264af6d39ee'

// Load your client certificate and private key
const cert = fs.readFileSync('cert.pem');  // Path to your client certificate
const key = fs.readFileSync('key.pem');    // Path to your client private key

// create JSON payload
let data = JSON.stringify({
  "row": {
    "fieldKey": "fieldData"
  }
});

// Create a custom HTTPS agent with the certificate and key
const httpsAgent = new https.Agent({
    cert: cert,           // Client certificate
    key: key,             // Client private key
 });

let config = {
  httpsAgent: httpsAgent,
  headers: { 
    'Authorization': 'Certificate dXNlcm5hbWU=', // authorization header contains base64 encoding of "username"
    'Content-Type': 'application/json'
  }
};

axios.post(url, data, config)
.then((response) => {
  console.log(JSON.stringify(response.data));
})
.catch((error) => {
  console.log(error);
});

using System.Security.Cryptography.X509Certificates;

// include the certificate .pfx file
var certificate = new X509Certificate2("cert.pfx");
// use the certificate authentication url
string url = "https://cluster-mtls.technolutions.net/slate-university.test.technolutions.net/manage/service/import?cmd=load&format=11c409f5-7bc5-4cf2-bbb9-1264af6d39ee";
var handler = new HttpClientHandler();
handler.ClientCertificates.Add(certificate);
var client = new HttpClient(handler);
var request = new HttpRequestMessage(HttpMethod.Post, url);
request.Headers.Add("Authorization", "Certificate dXNlcm5hbWU=");
var content = new StringContent("{\n    \"row\": \n    { \n        \"fieldKey\": \"fieldData\"\n    }\n}", null, "application/json");
request.Content = content;
var response = await client.SendAsync(request);
response.EnsureSuccessStatusCode();
Console.WriteLine(await response.Content.ReadAsStringAsync());

import requests
import json

# use the Certificate Authentication URL
url = "https://cluster-mtls.technolutions.net/slate-university.test.technolutions.net/manage/service/import?cmd=load&format=11c409f5-7bc5-4cf2-bbb9-1264af6d39ee"

# Add certificate and private key paths to pem files as appropriate
cert = ('cert.pem','key.pem')

#build a JSON payload
payload = json.dumps({
  "row": {
    "fieldKey": "fieldData"
  }
})

# add the request headers
headers = {
  'Authorization': "Certificate dXNlcm5hbWU=",
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload, cert=cert)

print(str(response.status_code) + ": " + response.text)

curl --cert cert.pem --key key.pem --location 'https://cluster-mtls.test.technolutions.net/slate-university.test.technolutions.net/manage/service/import?cmd=load&format=11c409f5-7bc5-4cf2-bbb9-1264af6d39ee' \
--header 'Authorization: Certificate dXNlcm5hbWU=' \
--header 'Content-Type: application/json' \
--data '{
    "row": 
    { 
        "fieldKey": "fieldData"
    }
}'
echo "\nthis one is for greybeards who like Unix"