Formats for Managing Data Flows

We were discussing a new web based intranet project with a customer this other day. This customer has an existing system that needs upgrading and has export facilities.

When we reviewed their operation it one of the key steps was to take data generated from their legacy system (call it system 'A') and review it before it was imported into System 'B'.

Without going into detail, issue and the obvious benefits of overcoming the manual export, check and import process it was a problem for them in another way. You see the tool that was easiest for them to use to check the data was MS Excel and so the data was exported in CSV format.

Unfortunately, certain fields in the exported data contained commas, line-breaks and other nasty characters which means that the format which arrived was very cumbersome to analyse. It also caused a problem when importing into system B.

Understanding that there are right and wrong data formats is key to dataflow management and in this case our approach is to make three significant changes that will improve the data flow process and save them time and effort.

Firstly, system A is going to have a web interface for their checking process. This does away with the need to use MS Excel and allows us to programme in additional safeguards such as change log and rollback on any data that they modify.

Secondly, when exporting is required to system B this is done via XML not CSV format. XML is much more robust in allowing markup and other things that may well break a CSV format. It is also a common format understood by their system B during the import process.

Thirdly, by the application of rules the manual process is being semi-automated which - although a manual check still takes place - is a lot less onerous and time-consuming that it was.

Needless to say, they're happy that they called us in and we're looking forward to finalising the project over the next few months.

If there's a dataflow issue that you'd like Emerging Innovations to help you with - either consultatively or as a developer - please don't hesitate to give us a call and we can have a chat.