Creating a Custom Source Format
  • 09 May 2024
  • 9 minute read
  • Dark
    Light
  • PDF

Creating a Custom Source Format

  • Dark
    Light
  • PDF

Article summary

A source format is a collection of standardized settings and criteria for a dataset uploaded frequently from the same source. The source format defines the destination fields and values in Slate and allows users to prescribe consistent import behavior, like:

  • New record creation,

  • Updates to existing applicant records

  • De-duplication within a source file

Source formats automate data import by configuring standard Slate SFTP options.

Best Practice: Use the test environment

Test data imports in the test environment before importing data in the production environment. Set up the import in the production environment, refresh your test environment, and activate the import in the test environment to make sure results are as expected before activating the imort in production.

Five steps to creating a new source format:

  1. Create a new source format, either from scratch or from a previous upload.

  2. Configure the General tab, which determines basic settings.

  3. Configure the Format Definition tab, which instructs Slate how to interpret incoming data.

  4. (Optional) Configure the Import Automation tab.

  5. Configure remap settings by uploading a sample file.

Step 1: Creating the source format

You have two options when creating a new source format:

  • Generate the source format using settings from a previous upload

  • Create the source format from scratch

Generating a source format from a previous upload

  1. Select Database on the Slate navigation bar and select Sources / Upload Dataset from the Import and Data Processing section.

  2. Select a previous ad hoc import to use as the foundation for the new source format.

  3. Select Generate Source Format. Follow the remainder of steps in this article to configure.

    Select_Generate_Source_Format

Creating a new source format from scratch

To create the new source format from scratch:

  1. From the main navigation, select Database.

  2. Under Import, select Source Formats.

  3. Click New Source Format. The New Source Format popup appears.

Step 2: Configuring the General tab

On the General Tab, fill in the information that describes your source format. Each field is explained below.

New Source Format - General Tab Default Configurations

Setting

Description

Status

Select Active to make the source format available for use in Upload Dataset.

Name

Give the source format a descriptive internal name.

Format

Enter the imported file format for display purposes only. For example, Excel, CSV, or Fixed Width.

File formatting is defined on the Format Definition tab in the XML field.

Type

One-Time / Differential

Retains existing source files alongside new source. Most imports use this type.

Cumulative / Replaceable

Replaces any existing source files with a new source file each time. For example: the standard Checklist Import (Financial Aid), which includes every record every time the file is imported.

Remap As Of Date

For now, enter the current date.

This date can be changed to limit the data values available to the source format for remapping.

If, for example, a file is imported with no header row, and first row data become the headers, enter a remap-as-of date after this erroneous file was imported.

Remap Active

Leave this as Inactive until you have configured remap settings (step five).

The remap-as-of date must be populated to activate this setting.

Scope

Leave this setting set to Person/Dataset Record unless the source format imports Slate users.

Dataset

Select a dataset into which the source format should import data.

Unsafe

If a source format is Unsafe, it can update person-scoped data on student records with an active application.

Select Unsafe only if you trust the source to contain accurate, up-to-date information. An import into Slate from your student information system is an example of a source format that you might set as Unsafe.

Purchased lists of student information are examples of imports that should have Safe source formats.

Hide

Determines whether the import will create source interactions on the affected record. You'll usually want to Create source interactions to display activity history on a record's timeline.

Daily or frequent data updates/imports can contribute to the creation of many rows of data interactions on records. If the source format will be used for a daily feed from an SIS or other application, select Hide source interactions

Disable Update Queue

Allow records to enter update queue upon import (allow rules to fire)

  • Records imported or updated from this source format enter an update queue.

  • This queue runs deferred rules and adds records to queues for those data exports that are configured to only include records updated since the query last ran.

Prevent records from entering update queue upon import (disables rules from firing)

  • Select this option if the source format is a backfeed that sets irrelevant information in the context of query update queues and rules.

  • This prevents a feedback loop wherein each updated record is exported from Slate into an external system, which then exports data for each of those updated records back to Slate.

  • If the update queue were enabled, those records in that import would be included in the export again the next time it ran, even if the import was the only update on those records.

Update Only

This tells Slate how to handle records in the incoming dataset that do not match records within Slate.

Allow record creation

Slate will create a new record for any records in the incoming dataset that do not match records in Slate.

Update only

Slate will not create new records, and the dataset will only update records on which it matches.  Note that when Update Only is selected for an application data import, an application will not be created even if the person record already exists.

Dedupe Records

By default, Slate will create a new record for every row when Allow record creation is selected and an existing match cannot be found.  By selecting Dedupe records, Slate will evaluate the source file and dedupe records in the source based on an exact match of First Name, Last Name, Email, and Birthdate.

Notification and Notification Email

To receive email notification if an imported file associated with the Source Format fails, select Failures only and enter the email address(es) for notification. Select Successes and Failures to be notified of both successful and failed imported files for the Source Format.

Read Permission and Upload/Build Permission

Permission to the source format can be limited to specific permissions. 

Step 3: Configuring the Format Definition tab

New Source Format - Format Definition Tab

The settings on the Format Definition tab provide Slate with specific instructions for interpreting the incoming data.

Setting

Description

XML

Use XPath syntax to tell Slate how to parse the data source. For example, you can define whether the file is delimited, if there is a header row, and what file format is used.

See the Common XML Examples article for specific examples that can be copied and pasted into this field.

SQL

For most data imports, the SQL text area should be left blank. Consider creating a community forum post to determine if your case requires custom SQL.

Source Index

Leave this set to the default Use modern cloud storage unless you are working with an existing source format that requires the legacy source.index table.

PGP Private Key

Paste the private key from the PGP software.

PGP Password

Each PGP private key is typically encrypted with a password.

📖 Further Reading

Format Definition XML Examples

Step 4: (Optional) Configuring the Import Automation tab

New Source Format - Import Automation tab

The settings on the Import Automation tab are optional and will not apply to every source format.

Setting

Description

Import Path/Mask

Include the file name pattern that Slate should look for to know that the file should be imported using this Source Format.

Best Practice: Append file names with the date

Slate retains files for a period of time in a Processed directory within the /incoming folder. If two file names are identical, the second file load deletes the first from the Processed directory. Appending file names with the date prevents this.

Example import from FTP mask field: SIS_to_Slate_*.txt

Slate automatically imports any file in the /incoming directory that contain that file name pattern, including: SIS_to_Slate_09032013.txt

Import Remote Server

Leave this field blank unless the data for this source format will be delivered via an API call from Slate or an external SFTP server.

API Calls

Include the full URL, including https:// and any necessary query string parameters.

Remote SFTP Servers

The syntax (including support for remote service account users) is the same as when configuring a remote server for data exports.

HTTP Headers

Add a list of headers in the form, for example: VALUE

Import Order

(Optional, if import path/mask exists) Specify an import order.

For example, if you have an application file and a supplement file delivered to the SFTP directory together, and you want the application file to be processed prior to the supplement, enter 1 in the application's source format and 2 in the supplement's.

Import Frequency

Remote SFTP

  • By default, Slate sweeps the SFTP directories for new files that match the specified import path/mask every 15 minutes.

  • Decrease frequency by entering the number of minutes between file sweeps, for example, 1440 minutes for a once-daily import.

  • The import frequency setting is limited to source formats picking up from remote SFTP servers and web services—not the Technolutions SFTP. 

Web Service

  • Slate's remote import automation polls the end point every 24 hours by default.

  • Adjust the import frequency up to hourly by setting the frequency value to 60 minutes; any value less than 60 will poll the endpoint hourly.

Web Callback URL

Enter a webhook URL to send a Web Service HTTP POST to an external system. This POST includes a JSON Payload and confirms that the import was successful in Slate. The external system receiving this confirmation can use it to mark its records as exported and updated in Slate.

Below is an example of a POST. For any records not updated or exported, a process can be implemented within your institution that prompts a member of your team to review and determine the root cause.

  {
    "status": "success",
    "source": "d13a8c60-d6da-2844-4704-0d3d78eed25f",
    "summary": "Common App Prospects",
    "database": "XYZ",
    "started" : "2022-06-28T06:12:22.345Z",
    "stopped" : "2022-06-28T06:12:23.577Z"
   }

When you are satisfied with the source format settings, click Save. You are redirected to the source format's overview page.

Step 5: Defining remap settings

☑️ Prerequisite Skills

Upload Dataset Stages

With a source format created, upload a sample file to define remap settings. Ensure a remap-as-of date has been set.

🔔 Important!

The data import and merge processes in Slate cannot be undone. Exercise extreme caution when uploading datasets of any kind, particularly for predefined formats like score reports. Before uploading, always confirm the dataset you're uploading is consistent with the format type you have selected and with past datasets of this type.

  1. On the overview page for source format you've just created, click Upload File(s).

  2. Select a sample file to upload. The sample file should exactly match the format that you intend to send Slate. Slate processes the file. When the upload is complete, the status on the Upload Dataset page changes from Pending to Completed or Awaiting Activation.

  3. Click the name of the source format. You are redirected to the remap stage, where source fields can be mapped to destination fields within Slate. See the article Upload Dataset Stages for more information on this step.

  4. When you've completed remapping, click the name of the source format in the breadcrumb menu to return to the source format overview page.

  5. Click Edit. A pop-up appears.

  6. In the General tab, set Remap Active to Active.

  7. Click Save.

  8. Slate processes any files that were uploaded on or after the remap-as-of date using the remap settings that you defined previously.

Any additional files that are uploaded to Slate using the new Source Format will also use the Source Format and remap settings that you configured using these steps.

Awaiting import through Source Format remap

If you've received this error, be sure that your source format has the appropriate XML setting on the Format Definition tab. Use one of the examples in the Format Definition XML article to build it out.


Was this article helpful?