Handling huge text files (which are either have extension 'csv' or fixed length) is a challenge in CPI (SAP Cloud Platform Integration) for processing. In CPI usually, scenarios comes where requirement is to process an input csv or any other character delimited text file. These files are of huge size compared to when we get data as xml or json format.
Mostly before converting data to xml format that is required for mapping, we do read them via groovy scripts to manipulate the data and we also need to convert incoming data into string format. Here, during converting into string format leads to memory issues.
In this blog post, the purpose is to provide alternate ways to handle them, not only how to read large files but also how to manipulate the data in required format.
The input file data can have “,” or tab or “|” delimiters or is of fixed length, which creates additional complexity as first data need to be read, sorted, converted to XML (for mapping to target structure format). Also, most of the time we have to perform some validations steps beforehand, to avoid processing of invalid data.
Conclusion: This blog is helpful to build required groovy scripts in SAP Cloud Platform Integration Flows. There are multiple cases, need to be handled for large text files in csv or in another delimited format, which requires reading the entire file. These types of business requirement can be achieved with the solution given in this blog.