Processing the NLPG (part 1)

Here's the plan for processing the NLPG update file every night:

1. File is grabbed from NLPG FTP server and put onto a location on a server on our LAN. This might be something we can do with Biztalk or we might have to find another way.
2. The file will need debatching and its data inserting into a series of seperate tables. Biztalk can automate this process - it can pick up the file, put its contents into a string object, pass this object to the web service that I have written for debatching and get a success/fail message back to report on.
3. Now I have four tables containing the information that I need. I can run my filtering script against these to produce a collection of unique UPRN's that have changed since the last update (or are new inserts) and I need to pass these into the data hub.
4. The changes / updates will be fed into an 'NLPGChanges' table and picked up by the Biztalk SQL Adapter. This will require a custom schema to be written for the NLPG data as defined by my database script.
5. Biztalk will process each individual update / insert, convert it into hub - compliant XML and pass into the hub via the Biztalk adapter.

That's the plan so far.