Archive

Author Archive

FixedWidthStreamReader, Conceived from Death

October 1, 2014 1 comment

The other day, I had the task of converting an function that calculates annuity factors from VBA to C#.  A couple of the function parameters were the desired Male and Female mortality tables.  Thus the pun with the post title (…From Death).

tl;dr – Jump to the code to see how I used Generics, Compiled Expression Trees, and Convert.ChangeType to create my FixedWidthStreamReader.

Given the desired tables, the old VBA code would read in all the rates from text files…120 rows and 2-3 columns in fixed width format.  Not being expert in file streams, I hit Google and ended up at the Stack Exchange post How to read fixed-width data fields in .Net.  You can read the post and the comments, but essentially, the original post required caller to call the following method for each ‘property’ they wanted read.  Additionally, you had to call the properties in the correct order as they appeared on the line since you didn’t pass a position in, but just a size that the reader internally used to keep track of the position.

T Read<T>( int size );

One of the comments suggested mimicking the StructLayoutAttribute which made sense as it keeps the file layout definition neatly packed away as attribute decorations on the output class instead of requiring each client of the reader class to know this definition.

The original poster never changed his code to use attributes and then I received an email from another user who commented on that post (Thanks Giuseppe) and posted his own implementation on the Stack Overflow post Reading data from Fixed Length File into class objects.  Given that inspiration, I decided to update the code with the given goals:

  1. The read<T>( T data ) method immediately jumped out to me as something that I thought I could simplify using Convert.ChangeType.
  2. I wanted to be able to read all the lines of a file and return a list of desired T objects. 
  3. It seemed that if you wanted to read more than one line in the original SO solution, that you would have to manage the parent Stream independently of the FixedLengthReader stream.
  4. I didn’t want to have to have the caller have to instantiate a T object and pass it into the reader.  I wanted the reader to return null or T objects appropriately.

So below is my attempt at a solution.  I derived from StreamReader, but since I really only intend the caller to use ReadLine or ReadAllLines, it may be better to change to just have a private StreamReader inside class.

Additionally, I'm no expert at streams, so not sure how much more performant the Stack Exchange solution’s ReadToBuffer is versus just a simple ReadLine, but here is my code.

FixedWidthStreamReader Code Snippets

Given file of: 
Person 1          1973 100000.54 
Grandparent 1     1950 2000000 

 

This is my type T class that each row in a file represents.  Each field that I want read in is decorated with a Layout attribute describing the start position and the length>

class PersonalInfo
{
   [Layout(0, 20)]
   public string Name;
   [Layout(20, 5)]
   public int YOB;
   [Layout(24, 15)]
   public double Pay;

   public override String ToString() {
       return String.Format("String: {0}; int: {1}; double: {2:c}", Name, YOB, Pay);
   }
}

LayoutAttribute class (note, I should probably support Properties in addition to Fields, but that would be an easy fix).

[AttributeUsage(AttributeTargets.Field)]
class LayoutAttribute : Attribute
{
   public int Index { get; private set; }
   public int Length { get; private set; }

   public LayoutAttribute( int index, int length )
   {
       Index = index;
       Length = length;
   }
}

I will paste FixedWidthStreamReader code below, but here are three usage code scenarios.

1.  Read just the first line of file into a PersonalInfo object.

using( var sr = new FixedWidthStreamReader<PersonalInfo>( @"c:\users\terry.aney\desktop\fixed.txt" ) )
{
	var pi = sr.ReadLine();
	System.Diagnostics.Debug.WriteLine( pi.ToString() );
}

2. Read all lines of file into IEnumerable<PersonalInfo> object.

using( var sr = new FixedWidthStreamReader<PersonalInfo>( @"c:\users\terry.aney\desktop\fixed.txt" ) )
{
	foreach( var pi in sr.ReadAllLines() )
	{
		System.Diagnostics.Debug.WriteLine( pi.ToString() );
	}
}

 

3. Read all lines of a file one at a time into PersonalInfo objects (might want to do this if you want to stop reading when reach someone or something).

using( var sr = new FixedWidthStreamReader<PersonalInfo>( @"c:\users\terry.aney\desktop\fixed.txt" ) )
{
	PersonalInfo pi;
	while( ( pi = sr.ReadLine() ) != null )
	{
		System.Diagnostics.Debug.WriteLine( pi.ToString() );
	}
}


Finally, here is the FixedWidthStreamReader class.  I create compiled Lambda expressions for each Field on the object and call the compiled Action during ReadT()

public class FixedWidthStreamReader<T> : StreamReader where T : class
{
	private List<Tuple<int, int, Action<T, string>>> propertySetters = new List<Tuple<int, int, Action<T, string>>>();
	
	public FixedWidthStreamReader( Stream stream ) : base( stream ) { GetSetters(); }
	public FixedWidthStreamReader( string path ) : base( path ) { GetSetters(); }
	
	private void GetSetters() 
	{
		var myType = typeof( T );
		var instance = Expression.Parameter( myType );
		var value = Expression.Parameter( typeof( object ) );
		var changeType = typeof( Convert ).GetMethod( "ChangeType", new[] { typeof( object ), typeof( Type ) } );

		/* Should probably do Properties here too, if I change AttributeUsage for LayoutAttribute I would */
		foreach ( var fi in myType.GetFields() )
		{
			var la = fi.GetCustomAttribute<LayoutAttribute>();
			if ( la != null )
			{
				var convertedObject = Expression.Call( changeType, value, Expression.Constant( fi.FieldType ) );

				var setter = Expression.Lambda<Action<T, string>>(
					Expression.Assign( Expression.Field( instance, fi ), Expression.Convert( convertedObject, fi.FieldType ) ),
					instance, value
				);
				
				var prop = setter.Compile() as Action<T, string>;
				propertySetters.Add( Tuple.Create( la.Index, la.Length, prop ) );
			}
		}
	}

	public new T ReadLine()
	{
		if ( Peek() < 0 ) return (T)null;
		
		return ReadT( base.ReadLine() );
	}
	
	private T ReadT( string line )
	{
		if ( string.IsNullOrEmpty( line ) ) return null;
		
		var t = Activator.CreateInstance<T>();

		foreach( var s in propertySetters )
		{
			var l = line.Length;
			
			if ( l > s.Item1 )
			{
				s.Item3( t, line.Substring( s.Item1, Math.Min( s.Item2, l - s.Item1 ) ).Trim() );
			}
		}
		return t;
	}
	
	public IEnumerable<T> ReadAllLines()
	{
		string line = null;
		
		while ( !string.IsNullOrEmpty( ( line = base.ReadLine() ) ) )
		{
			yield return ReadT( line );   
		}
	}
}

 

Ultimately, I used this as a VERY SIMPLE learning experience on how to create compiled Expression Trees, which I still need to mess with more to fully understand, but hopefully this will give you both a simple introduction to that along with a pretty functional FixedWidthStreamReader.

Categories: C#, Expression Tree, Generics

Use SelectExcept When You Are Too Lazy To Type

February 27, 2013 Leave a comment

Do you love LINQPad?  Use it every day?  Do ad-hoc queries non-stop against your DataContexts?  Love C#/LINQ’s ability for anonymous object projections into new forms?  I’m assuming you said yes to all those.  Here is the question that drives this post…do you hate having to type *every* field *except* one (or a few) when you *only* need almost all fields?  If you answered yes to that last question or are wondering why I don’t just allow the entire row to be selected/returned, continue reading…

Read more…

Categories: C#, Extension Methods, LINQ

L2S Extensions…MVC style

February 9, 2012 1 comment

Four years later…I’ve finally come up for breath.  I hope to start a better pattern of trying to get some technical posts created once in a while.  Until I have my first inspiration – that I feel would be beneficial to anyone in the WWW, I thought I’d provide a little update to my L2S Extensions.

Here is what has changed since my last post 4 years ago.  I’m going to make the assumption that since there was really only three new improvements, that the code most have been pretty solid Smile.  Some of the changes below were due to the fact that the main code I generate for my company has shifted to the ASP.NET MVC framework and have started leveraging the System.ComponentModel.DataAnnotations namespace to decorate pretty much everything.  Since I was querying some of those same classes, it made sense to update my L2S Extensions code to obey some attributes as well.  All the changes below occur in DumpCSV().

Read more…

Categories: C#, Extension Methods, LINQ

I’ve Left Query Analyzer Hell For LINQPad Heaven

December 4, 2008 13 comments

So now that LINQPad has enabled intellisense SQL Server Management Studio, Query Analyzer, and even Joseph Albahari’s (LINQPad creator) own QueryEx have all been zapped from my memory.  I’ll no longer flounder in antiquated ANSI SQL, but instead flourish in fully typed C#/LINQ code.

But wait, there’s more, click to keep reading…

Categories: C#, Extension Methods, LINQ

LINQ to SQL Batch Updates/Deletes: Fix for ‘Could not translate expression’

April 20, 2008 12 comments

I’ve found and posted a new fix in the code from my original post: Batch Updates and Deletes with LINQ to SQL.  I’m not sure of the etiquette for this sort of thing: new post (like I’m doing) or just a comment in the original post.  But since I did get a fair amount of hits to the article but minimal comments, people who may have downloaded the code wouldn’t get an update notification and I want to be sure to make them aware of an issue/fix (assuming they are monitoring via a RSS feed).

But wait, there’s more, click to keep reading…

Categories: C#, LINQ

LINQPad: Preview DELETE and UPDATE SQL before it executes

April 18, 2008 Leave a comment

Like everyone else (or at least you should be), I use LINQPad throughout the day non-stop.  Primarily for database maintenance that would have previously been done by saving a *.sql script.  However, at first when I was new to LINQ (not that I’m a complete expert now), I’d get hesitant when I was about to perform a bunch of updates and/or deletes against my production data.  Being more comfortable in Transact SQL than in LINQ, I wanted to see the SQL statements that would execute before actually calling SubmitChanges().  As plenty of posts have mentioned, it is far more than simply a LINQ to SQL execution tool, but rather:

And LINQPad is more than just a LINQ query tool: it’s a code snippet IDE. Instantly execute any C# 3 or VB 9 expression or statement block!

- Joseph Albahari (creator of LINQPad)

With a couple of extension methods, you can do exactly that: preview the SQL before it executes.

But wait, there’s more, click to keep reading…

Categories: Extension Methods, LINQ

Batch Updates and Deletes with LINQ to SQL

April 14, 2008 32 comments

A couple weeks ago, I read the article, LINQ to SQL Extension: Batch Deletion with Lambda Expression by Jeffrey Zhao.  In case you didn’t read the article, it discusses the downside of most O/R Mapping frameworks when it comes to multiple updates or deletes.  He states the fact that a SQL statement for each row flagged as update/delete in the entity set is created.  I went about implementing something similar to what Jeffrey envisioned and I’ll explain some of the hurdles I had to overcome to achieve it.

But wait, there’s more, click to keep reading…

Categories: C#, Extension Methods, LINQ
Follow

Get every new post delivered to your Inbox.

Join 164 other followers