Batch API Overview

The Batch API allows you to upload held account data records in properly formatted and encrypted CSV files via SFTP. MX can provide a response file for each batch file submitted.

Check Files

Ensure your batch files are correctly formatted before submitting them to MX for processing by using our Batch File Checker:

1. Download the Batch File Checker, unzip it, and open csv-checker/index.html.
2. Follow the instructions presented on the webpage.

Use the sample CSV files included in the zip as guidance for file formatting and naming schema.

File Format

Batch files for creating users, members, accounts, and transactions must adhere to the following standardized format:

• Encoding: UTF-8.
• Format: CSV.
• Quoting: Use field quoting when necessary.
• Header Row: Required to specify the fields you're providing. For which fields are required/optional for each file, see File Contents.
• Customization: Line ending, separator, and quote characters are customizable.

Initial SFTP Setup

To set up access to MX's integration and production SFTP servers:

1. Provide source and destination IP addresses (and domain name, if applicable) for whitelisting.
2. MX will securely provide usernames and passwords for both integration and production environments, and port numbers.
3. (Optional) MX will provide public PGP keys for integration and production environments. All batch files must be encrypted with the environment-specific MX public PGP key.
4. (Optional) If you require response files, provide MX with:
   • SFTP URL and login information for your test and production servers.
   • Destination IP addresses for your SFTP server.
   • (Optional) Your public PGP key. If needed, you can provide separate keys for test and production servers.

Sending Files via SFTP

To send batch files to MX's SFTP server:

1. Prepare each file, ensuring proper formatting, file-naming requirements, and optional encryption with the environment-specific MX public PGP key.
2. Log into the MX SFTP server.
3. Navigate to the upload directory (default: files folder).
4. Upload files using SFTP commands.
5. If requested, MX will upload encrypted response files (using your provided PGP key, if applicable) to your SFTP server.

SFTP Hostname and Port

The MX SFTP hostnames for each MX environment are as follows:

• MX Integration Environment: int-sftp.mx.com
• MX Production Environment: sftp.mx.com

MX uses a non-standard port, which will be provided with your credentials.

Identifiers

Users, members, accounts, and transactions use identifiers provided by your system. This enables retrieval via other MX APIs using your system's identifiers.

We recommend identifiers consist of letters, numbers, dashes, and underscores, and are less than 1,024 bytes. Common choices include 128-bit UUIDs or 64-bit sequential integers.

Date and Times

Batch API resources often include _at and _on fields. Provide dates using either format, but not both:

• _at: Unix datetime.
• _on: ISO 8601 dates (for example, YYYY-MM-DD). Zero pad the month and day.

File Naming and Size

Batch files must follow a fixed naming convention. Each batch filename will have four portions plus the filename extensions. A manifest filename has three portions plus the filename extensions. Files that are not named correctly will not be processed.

Filename Format:

• Batch files: PREFIX.TYPE.DATE.SUFFIX.csv.pgp
• Manifest file: PREFIX.Manifest.DATE.csv.pgp

Batch File Naming Convention

Portion	Description	Example Value	Required?
PREFIX	Groups files into a batch for a given date. Must be the same for all files in a batch; unique with DATE across all client batches.	Bank, PreloadGroup	Yes
TYPE	Object type (singular). Must be one of the allowed types; Manifest for manifest files.	User, Member, Account, Transaction, Manifest	Yes
DATE	ISO 8601 date (YYYY-MM-DD or YYYYMMDD). Must be the same for all files in a batch.	2017-02-01	Yes
SUFFIX	Differentiates files of the same type within a batch. Increment for multiple files of same type; omit for Manifest files.	Split001, part001, Complete	Yes (except Manifest)
.csv	File extension. Always required.	.csv	Yes
.pgp	File extension for PGP-encrypted files. Required if file is PGP-encrypted.	.pgp	Conditional

Full Format:

• PREFIX.TYPE.DATE.SUFFIX.csv.pgp
• Manifest: PREFIX.Manifest.DATE.csv.pgp

Filename Examples:

• Bank.User.2016-12-31.Split001.csv.pgp
• BackfillProject.Member.2017-01-20.Split002.csv.pgp
• Credit.Account.2017-02-01.Part01of01.csv.pgp
• MyBank.Transaction.2017-02-01.Complete.csv.pgp

Batch Definition

A batch is a group of batch files that are intended to be processed together. The batch is identified by the combination of the PREFIX and DATE fields in the filenames. Every file in the batch must have the same PREFIX and DATE values, and the combination of those values must be unique across all batches for the client. If multiple batches are uploaded with the same DATE value, they must have different PREFIX values to be considered separate batches.

Example filenames for a batch file group containing one each of user, member, and account files, two transaction files, and the manifest file:

_10

PreloadGroup.Manifest.2017-10-31.csv.pgp

_10

PreloadGroup.User.2017-10-31.batch001.csv.pgp

_10

PreloadGroup.Member.2017-10-31.batch001.csv.pgp

_10

PreloadGroup.Account.2017-10-31.batch001.csv.pgp

_10

PreloadGroup.Transaction.2017-10-31.part001.csv.pgp

_10

PreloadGroup.Transaction.2017-10-31.part002.csv.pgp

Prefix

The PREFIX groups files from the same DATE into a batch, ensuring sequential processing according to the MX Architecture hierarchy (for example, users → members → accounts → transactions).

For example, a transactions file named Bank.Transaction.20170202.Split001.csv.pgp will not be processed until the user, member, and account files prefixed with Bank and having date 2017-02-02 have completed processing.

In contrast, independent data sets for a particular DATE can be given a different PREFIX value. MX will consider that set of files a different batch because of the different PREFIX. A batch using the same 2017-02-02 date as in the example above, but using Backfill as its prefix, won't wait for the batch with Bank as its prefix to complete before it starts processing. The batches will be processed in parallel.

Type

The TYPE indicates the object type and must be one of: User, Member, Account, Transaction, or Manifest. Use the singular form only.

Date

The DATE must be an ISO 8601 date with a four-digit year and zero-padded month and day (YYYY-MM-DD or YYYYMMDD). Note that this date represents the data itself, not necessarily the date of transmission. While it often coincides with the upload date for daily processing, it should reflect the historical date when sending backfill data. This DATE must be consistent for all files within the batch.

Suffix

The SUFFIX differentiates unique files of the same type within a batch. For example, if multiple Transaction files are being included in the batch, the SUFFIX allows them to be uniquely named. A recommended practice is to use a SUFFIX ending with a number and incrementing the number for each file of that type (part001, part002, and so on). A file of type Manifest does not require a suffix.

File Size

• Recommended: 1 million records or fewer for optimal performance.
• Maximum: 3 million records. Larger datasets must be split.

Upload Order

Objects must be created following the MX Architecture hierarchy. Group batches appropriately and upload them in the correct order to avoid errors. For example, transactions can't be created before the user, member, and account records have been created.

Early integrations that did not use manifest files had to ensure files were uploaded in strict hierarchy order to ensure files in the same batch were processed in the correct order.

Manifest files (required for all new integrations) ensure all files for a batch are received before processing starts, and ensure they are processed in the correct order.

When doing an initial data load, it is sometimes easier for a client to use separate batches to create each object in the MX hierarchy rather than trying to create them all in a single batch. In such a case, the batches must be uploaded in strict MX hierarchical order to ensure parent objects are created before child objects that attach to them. The order is:

1. User files
2. Member files
3. Account files
4. Transaction files

This ensures hierarchical objects are created in the correct sequence.

Manifest Files

Manifest files are required for all new integrations. The manifest lists all files that MX should expect to receive for that batch and allows optional checksums for each file, guaranteeing correct processing and data integrity.

It is a best practice to send the manifest file first. MX will begin processing the batch only when all listed files are received, and no files will be processed that aren't listed in the batch manifest.

Manifest File Naming

Manifests follow the same naming pattern: PREFIX.TYPE.DATE.csv.pgp. Use the PREFIX and DATE values for the batch, set TYPE to Manifest, and omit the SUFFIX.

Example: Bank.Manifest.2016-12-31.csv.pgp

Manifest File Contents

Manifests must be in CSV format. The filenames must be listed exactly as they will be sent to MX, including filename extensions. If the batch is encrypted, each batch file must be PGP encrypted and the manifest file itself must also be PGP encrypted and include the .csv.pgp extensions.

Two columns are supported:

Field	Description	Required?
filename	The name of the file in the batch (including extensions).	Yes
checksum	The SHA256 checksum of the unencrypted file.	No

An example of manifest file contents follows.

_10

filename,checksum

_10

Bank.User.2020-04-13.Split001.csv.pgp,389438d0f857041757a754e696375dc2b8089f2423ccbe5e3ff2a0e60ac1544f

_10

Bank.Member.2020-04-13.Split001.csv.pgp,89fbc47675603079be7f9feca7625d8ac38802ac3e2c492edeed04a327db8015

_10

Bank.Account.2020-04-13.Split001.csv.pgp,5474f58d52395952268b10342b45dff780a0039636aba3a21a0b0ad5775d062d

_10

Bank.Transaction.2020-04-13.Split001.csv.pgp,b423960ce42eadd02ab36f3ba9313d5ac2d533ba3a4d2266d15a2df43d46e0c1

Note that the checksum, when used, is calculated for the unencrypted file.

Number Limits

Decimal numbers are defined by precision,scale. Precision is the total number of digits; scale is the number of digits after the decimal. For example, 538.46 has a precision,scale of 5,2.