Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/dataxflo/public_html/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340
Drowning in Text Files – dataflow analytics blog Press "Enter" to skip to content

Drowning in Text Files

Lars Fiedler 0

Everyone has them laying around their computer somewhere: folders full of loosely formatted files.  It could be text, xml or csv data that was created for that one time use that is now becoming multiple use.  The original person and program that created them no longer exists, but you still need to modify the data in these hundreds of files for a new analysis.   In addition, these files might be scattered into different folder structures and with different names.  The thought of copy and pasting hundreds of small files together is a no-go but similarly is putting in a ticket for someone else to help you with a task.  Simple tasks like these are one example where Composable Analytics shine, offering the power to manipulate many files in an easy to use interface that is can be saved and stored for later use and shared with next unlucky soul that might need to use these files.



The example above was made one day when I received a request to modify a large number of xml files with new header data.  The person who requested it wanted me to write a bash script to do the modification for them when I pointed out this is a great use case for Composable.  Above is the example app I sent them as to how to do it.  You point the app to a directory with a list of .txt files in it, it goes through each one, reads the data, appends the header information, and saves it as a new file in another directory for you.  It’s clear to everyone what’s going on, it’s easy to modify with more complex logic, and its saved.

Lars Fiedler

Lars has comprehensive expertise building large complex software systems, and has served as a Software Engineer at MIT’s Lincoln Laboratory from 2010-2014, where he began developing Composable Analytics. Prior to joining Lincoln Laboratory, Lars worked as a Software Engineer at Microsoft Corporation from 2006 to 2010. Lars received his MS in Computer Science from Georgia Institute of Technology in 2004, and his BS in Computer Science from Georgia Tech in 2003.
Lars Fiedler

Leave a Reply

Your email address will not be published. Required fields are marked *