Monday, January 07, 2008

Parse large XML files

Our batch is one of the luckiest batch @ Computer Science & Engineering Dept of University of Moratuwa to have Sanjiva Weerawarna as a lecturer to teach XML and Web Services. That time only he returned from US to Sri Lanka and had worked for IBM from Sri Lanka. We were the first batch of him @ Uni of Mora. He was instrumental in putting up the solid foundation of XML related technologies in students mind and I can still remember his lecture about DOM parser and SAX parser for XML parsing.

OK, now we will come to the story….

One of my friends had problem in loading big XML file and wanted a solution to parse and then insert the records to the database for one of his application. He tried with XMLDocument and got the “Memory out of Exception”. Here we need to use the SAX parser instead of DOM parser and.NET XmlTextReader class comes to solve the problem which provides a fast, non-cached, forward only read access to read XML data. Using the XmlTextReader we can load large files 4kB chunk by chunk into memory and process.

You can read about working with large xml files here.
http://msdn2.microsoft.com/en-us/library/aa302289.aspx
http://msdn2.microsoft.com/en-us/library/ms998559.aspx

Labels: ,