Sunlife: Two File Processing and Single File Processing

Owned by Rajeev Rai
Last updated: Sept 04, 2024
3 min read

Objective:

  • Improve File Processing System

  • Two-File Processing Validating & Merging

  • Single-File Processing Validating

  • Dashboard Implementation

Overview:

Sunlife wants to improve its system for processing files, with two main types of processing:

  1. Two-File Processing: When the team receives two files, they will update the procedures for checking and validating the data. Additionally, they will introduce a new process of connecting to Sunlife’s system to obtain a unique identifier (similar to a tracking number). Furthermore, they will consolidate individual reports into a single document and enhance the clarity of error reports. (After validating it will merge and single o/p file would be created)

  2. Single-File Processing: For scenarios involving the receipt of a single file, a new system will be established to examine the file, address any issues, and route it to the appropriate destination. Additionally, the system will ensure timely notification of relevant individuals in case of any errors.(I/p file from any partner it will convert into standard format validation check, after this o/p file would be generated and placed at the target location)

Additionally, they need a simple dashboard (like a control panel) to show the progress of files being processed. This will make it easy for users to see if the files are done, have errors, or are waiting in line.

Object Zip:

Import the object in a replicated environment or Training Environment

click here to DOWNLOAD

Solution Design:

The diagram depicts a data processing workflow where clients and trading partners send data in various formats to be standardized, validated, and output in specific formats. Below is the step-by-step breakdown of the workflow:

1. Input Stage (Any Format)

  • Clients and Trading Partners:

    • Different clients (e.g., Client_GERDAU, Client_ABC, Client_XYZ) and trading partners (e.g., TP_FF6000, TP_MM365, TP_N000) send data in different formats such as XML, CSV, or JSON.

    • The data is received in client-specific or partner-specific layouts.

2. Transformation (Adeptia)

  • Adeptia Transformation:

    • This transformation tool converts the input data into a standard client layout for clients and a standard trading partner layout for trading partners.

    • These standardized layouts help unify the incoming data into a consistent format for further processing.

3. Validation and Merging (Adeptia)

  • The standardized data goes through another stage of validation and merging.

    • Validation: Ensures data correctness and conformity to rules defined in the standard layouts.

    • Merging: Combines the standardized data into a merged dataset, while segregating any bad data or discrepancies.

4. Output Stage

  • Standard Output Layouts:

    • The merged and validated data is formatted into a standard output layout, ready for export in a specified format (e.g., CSV).

    • This merged output file is typically sent via FTP to the intended destination.

  • Discrepancy & Error Handling:

    • Bad Data: Any data that failed validation is flagged and formatted into a standard data discrepancy layout.

    • Orphan Data: Data that cannot be matched during the merging process is classified as unmatched orphan data and is formatted into a standard orphan record layout.

5. Reporting Stage

  • Error and Orphan Records:

    • The bad data and orphan records are outputted in CSV format as either an error report file or an orphan report file.

    • These reports are typically sent via email to the concerned parties for review and corrective actions.

6. Final Outputs

  • Merged Output: Sent via FTP in CSV format.

  • Error Report & Orphan Report: Sent via email in CSV format.

This workflow ensures that data from various clients and trading partners is consistently processed, validated, and outputted in a reusable and standardized manner.