Archive

Archive for the ‘C#’ Category

MSBuild/Visual Studio Does Not Copy Nuget Package Assemblies

May 6, 2018 Leave a comment

tl;dr – Jump to the code to see how I used a custom MSBuild Task that scans all non-GAC’d references (binary and project) of a client/host application to copy all missing dependent assemblies from any level in a solution structure.

The Problem…

Actually, the title doesn’t sum up everything but I needed a catchy search title.  It is mostly true, but in reality, there are cases where standard assembly references to third party assemblies, or even projects I’ve created, aren’t copied either.

Here is my scenario.  I have a hundred plus client projects written against my company’s ‘framework assemblies’ that look like the following:

  • Client Web Application –> references –>
    • Framework Assembly 1
    • Framework Assembly 2 –> references –>
      • Nuget: Newtonsoft.Json
      • Framework Assembly 3 –> references –>
        • Framework Assembly 4 –> references –>
          • Nuget: Newtonsoft.Json
    • Framework Assembly 4 –> references –>
      • Nuget: Newtonsoft.Json    

As you can see, Framework Assembly 4 is a ‘utility’ assembly that can/is referenced at multiple levels in the project architecture.  Also, Newtonsoft.Json is referenced at multiple levels.  The problem is, when I build Client Web Application both Framework Assembly 3 and Newtonsoft.Json are not in the output directory even though all references in the project have CopyLocal=true.

Below is an image of a simple console application reproducing the same problem:

DependentCopySln

When you build ClientApplication, the output directory is as follows:

ReleaseBefore

Notice ClassLibrary3* and Newtonsoft* files are missing.  You can get the source for that sample solution here if you want to replicate the problem yourself.

My first bad solution…

My first solution to this problem was pretty poor.  I simply added the Newtonsoft.Json nuget package to the Client Web Application as well.  This worked for a bit, but there were three main problems…

  1. If the framework assemblies updated their Nuget package and the client application did not, I get a runtime errors due to version conflicts.
  2. If a developer who is not aware of this hack runs an extension like refassistant, they might remove the ‘unused Nuget reference’ from the client application, again experiencing runtime errors for missing assemblies.
  3. If a framework assembly adds another Nuget package assembly, I’d have to update all the client applications to add the new Nuget package as well – painful.

Correct solution, a custom MSBuild Task…

I finally decided to bite the bullet and implement the correct solution, which is to have a custom MSBuild task in the client application project that scans for and copies all required dependent assemblies.

A good walk through of creating a custom MSBuild task can be found at The Custom MSBuild Task Cookbook.  The only tip I would offer is that you need to compile the MSBuild Task project on the x64 Platform Target.  I can’t remember where I saw a mention of this, probably somewhere on Stack Overflow, but many of the walk through examples you’ll find on the net describing the creation of a MSBuild task don’t mention this and you get a confusing error when it tries to execute your task:

The "{task name}" task could not be loaded from the assembly {task assembly file}. Could not load file or assembly ‘{assembly name}’ or one of its dependencies. The system cannot find the file specified. Confirm that the declaration is correct, that the assembly and all its dependencies are available, and that the task contains a public class that implements Microsoft.Build.Framework.ITask.

However, when you review everything mentioned, everything seems in order.  But, simply flipping the Platform Target to x64 makes things start to work.

Building DependencyCopy task…

I came across Recursively Copying Indirect Project Dependencies in MSBuild which was almost what I was after.  My goal is almost identical, but the main problem I had with the solution found there is that all the code was in an inline code task and I would have to insert the 100 or so lines of code into the 100 plus client *.csproj files.  That wasn’t going to be acceptable.  I knew I needed this code in one central, debug-able location and as the author admitted,

Yeah, that’s a lot of code. And I could probably tighten it up, but I’m only using it once, in one place

So, I addition to tightening it up, I had to add one more feature.  My client projects (and my framework assemblies for that matter) do not always have project references.  Actually, I normally reference the assemblies so I don’t have to rebuild extra projects every time I publish a client.  So my code had to find binary references to assemblies built by my company and find dependent references of that project as well.  Because of that, I have some code that is specific to my company that I will present here first:

var btrBinaryReferences =
	project.Elements( ns + "ItemGroup" )
			.Elements( ns + "Reference" )
			.Where( r => ( (string)r.Attribute( "Include" ) ).StartsWith( "BTR." ) )
			.Select( r =>
				GetProjectFileFromBinaryAssembly(
					Path.Combine( projectFolder, (string)r.Element( ns + "HintPath" ) )
				)
			);

As I mentioned, any binary references to my company’s assemblies (files with names starting with BTR.) need to have the associated *.csproj scanned as well.  Once I identify these assemblies with the StartsWith() clause, I then get the *.csproj via GetProjectFileFromBinaryAssembly.  The code in that method has some documented assumptions applied based on BTR’s local file system structures.

Finally, before listing the code, I’ll show you the snippet of MSBuild script you need to insert into your *.csproj file.  Let’s assume you build the DependencyCopy task in a project named MSBuildHelpers.csproj with a namespace of MSBuildHelpers.DependencyCopy and that the MSBuildHelpers.dll assembly is placed in C:\Assemblies.  You’ll want to be running the DependencyCopy task after the build is finished so you would insert the following AfterBuild target right before the ending </Project> tag:

<UsingTask TaskName="BTR.Evolution.MSBuild.DependencyCopy" AssemblyFile="C:\BTR\Source\Assemblies.Evolution\BTR.Evolution.MSBuild.dll" />
<Target Name="AfterBuild">
  <DependencyCopy StartFolder="$(MSBuildProjectDirectory)" ProjectName="$(ProjectName)" Configuration="$(Configuration)" OutputFolder="$(OutputPath)" />
</Target>

After adding that and rebuilding ClientApplication, the following shows that ClassLibrary3* and Newtonsoft* files are correctly copied to the directory as well.

ReleaseAfter

DependencyCopy code…

Finally, the code.  Let me know if you have any questions or comments.  Hopefully this can help you out as much as it has been helping me.

	public class DependencyCopy : Task
	{
		[Required]
		public string StartFolder { get; set; }
		[Required]
		public string ProjectName { get; set; }
		[Required]
		public string Configuration { get; set; }
		[Required]
		public string OutputFolder { get; set; }

		XNamespace ns = "http://schemas.microsoft.com/developer/msbuild/2003";

		public override bool Execute()
		{
			Log.LogMessage( MessageImportance.High, $"Scanning {ProjectName}.csproj for BTR project references/assemblies..." );

			var projectsToScan = GetDependentProjectsToScan().ToArray();

			Log.LogMessage( MessageImportance.High, $"Scanning BTR projects for additional assemblies to publish..." );

			var assembliesToCopy = GetDependentAssemblies( projectsToScan ).ToArray();

			var assemblyFolder = Path.Combine( StartFolder, OutputFolder );

			Log.LogMessage( MessageImportance.High, $"Copy dependent assemblies to {assemblyFolder}..." );

			foreach ( var a in assembliesToCopy )
			{
				var assemblyFile = Path.GetFileName( a );
				var assemblyDestination = Path.Combine( assemblyFolder, assemblyFile.EndsWith( ".resources.dll", StringComparison.InvariantCultureIgnoreCase ) ? Path.GetDirectoryName( a ).Split( '\\' ).Last() : "", assemblyFile );

				if ( !File.Exists( assemblyDestination ) || new FileInfo( a ).LastWriteTime > new FileInfo( assemblyDestination ).LastWriteTime )
				{
					Log.LogMessage( MessageImportance.High, $"\tCopy {a} to {assemblyDestination}." );
					File.Copy( a, assemblyDestination, true );
				}
			}

			return true;
		}

		class DependentProject
		{
			public string AssemblyName { get; set; }
			public string ProjectFile { get; set; }
		}

		private IEnumerable<DependentProject> GetDependentProjectsToScan( string projectFile = null )
		{
			var isRootProject = string.IsNullOrEmpty( projectFile );
			var fileName = projectFile ?? Path.Combine( StartFolder, ProjectName + ".csproj" );
			var project = XElement.Load( fileName );
			var projectFolder = Path.GetDirectoryName( fileName );

			var projectsReturned = new HashSet<string>();

			Func<string, string> getUniqueProjectToScan = f =>
			{
				var projFile = new FileInfo( f ).FullName;

				if ( !projectsReturned.Contains( projFile ) )
				{
					projectsReturned.Add( projFile );
					if ( isRootProject )
					{
						Log.LogMessage( MessageImportance.High, $"\tProject reference: {projFile}" );
					}
					return projFile;
				}
				else
				{
					return null;
				}
			};

			var projectReferences =
				project.Elements( ns + "ItemGroup" )
						.Elements( ns + "ProjectReference" )
						.Select( r => new DependentProject { ProjectFile = new FileInfo( Path.Combine( projectFolder, (string)r.Attribute( "Include" ) ) ).FullName } );

			var btrBinaryReferences =
				project.Elements( ns + "ItemGroup" )
						.Elements( ns + "Reference" )
						.Where( r => ( (string)r.Attribute( "Include" ) ).StartsWith( "BTR." ) )
						.Select( r => (string)r.Element( ns + "HintPath" ) )
						.Select( r =>
							 new DependentProject
							 {
								 AssemblyName = new FileInfo( Path.Combine( projectFolder, r ) ).FullName,
								 ProjectFile = GetProjectFileFromBinaryAssembly( Path.Combine( projectFolder, r ) )
							 }
						);

			var allReferences = projectReferences.Concat( btrBinaryReferences ).ToArray();

			foreach ( var dependentProject in allReferences )
			{
				var projFile = getUniqueProjectToScan( dependentProject.ProjectFile );

				if ( projFile != null )
				{
					yield return dependentProject;

					// Scan for all project references in 'referenced projects'
					foreach ( var s in GetDependentProjectsToScan( projFile ) )
					{
						projFile = getUniqueProjectToScan( s.ProjectFile );

						if ( projFile != null )
						{
							yield return s;
						}
					}
				}
			}
		}

		private string GetProjectFileFromBinaryAssembly( string binaryAssemblyName )
		{
			/*
			* This function is specific to BTR's project structure and location of framework assemblies.
			* We have a structure like the following
			*
			* C:\BTR\Source\Assemblies.Evolution - any binary references to framework assemblies will be
			*										pointed at this directory.
			* C:\BTR\Source\Evolution\ - This is root folder of all framework assemblies.
			* C:\BTR\Source\Evolution\BTR.* - These folders are all the framework projects where the name
			*								   of the folder matches the name of the *.csproj file.
			*
			* Given that structure, the code below can find the *.csproj given a binary assembly found
			* in Assemblies.Evolution.
			*/
			var binaryAssembly = new FileInfo( binaryAssemblyName );
			var projectName = Path.GetFileNameWithoutExtension( binaryAssembly.FullName );
			var sourcePaths = new[] {
				binaryAssembly.Directory.Parent.FullName, // One level up from 'assembly' folder is source for older framework libraries
				Path.Combine(binaryAssembly.Directory.Parent.FullName, "Evolution")
			};

			var projectFileName =
				sourcePaths
					.Select( p => Path.Combine( p, projectName, projectName + ".csproj" ) )
					.First( p => File.Exists( p ) );

			return projectFileName;
		}

		private IEnumerable<string> GetDependentAssemblies( IEnumerable<DependentProject> projectsToScan )
		{
			var assembliesReturned = new HashSet<string>();

			Func<string, bool> tryAddAssemblyToCopy = f =>
			{
				var binaryReference = new FileInfo( f );

				var binaryPath = Path.GetDirectoryName( binaryReference.FullName );
				var binaryFileName = Path.GetFileNameWithoutExtension( binaryReference.FullName );

				var resourceFile = binaryFileName + ".resources.dll";

				var assemblies = new[]
				{
					new { IsDll = true, Assembly = binaryReference },
					new { IsDll = false, Assembly = new FileInfo( Path.Combine( binaryPath, binaryFileName + ".pdb" ) ) },
					new { IsDll = false, Assembly = new FileInfo( Path.Combine( binaryPath, binaryFileName + ".xml" ) ) }
		}.Concat(
					// Make sure to copy all resource files as well
					binaryReference.Directory.GetDirectories()
									.Where( d => File.Exists( Path.Combine( d.FullName, resourceFile ) ) )
									.Select( d => new { IsDll = false, Assembly = new FileInfo( Path.Combine( d.FullName, resourceFile ) ) } )
				).ToArray();

				foreach ( var a in assemblies )
				{
					if ( a.Assembly.Exists && !assembliesReturned.Contains( a.Assembly.FullName ) )
					{
						assembliesReturned.Add( a.Assembly.FullName );
					}
					else if ( a.IsDll )
					{
						// If the dll file doesn't exist, don't even check for the pdb/xml
						return false;
					}
				}

				// if made it this far, I at least added the dll
				return true;
			};

			foreach ( var p in projectsToScan )
			{
				var project = XElement.Load( p.ProjectFile );
				var assemblyDirectory = Path.GetDirectoryName( p.AssemblyName ?? p.ProjectFile );
				var projectDirectory = Path.GetDirectoryName( p.ProjectFile );

				// For the current BTR project, try to include the project assembly from the same Configuration
				// setting, otherwise falling back to the Release mode.  Using AssemblyName in case it was a binary
				// reference, want to grab the files from wherever that is stored.
				var projectAssemblyName = Path.GetFileName( p.AssemblyName ?? (string)project.Elements( ns + "PropertyGroup" ).Elements( ns + "AssemblyName" ).First() + ".dll" );

				if ( string.IsNullOrEmpty( p.AssemblyName ) )
				{
					if ( !tryAddAssemblyToCopy( Path.Combine( assemblyDirectory, "bin", Configuration, projectAssemblyName ) ) )
					{
						tryAddAssemblyToCopy( Path.Combine( assemblyDirectory, "bin", "Release", projectAssemblyName ) );
					}
				}
				else
				{
					tryAddAssemblyToCopy( Path.Combine( assemblyDirectory, projectAssemblyName ) );
				}

				foreach ( var h in project.Elements( ns + "ItemGroup" ).Elements( ns + "Reference" ).Elements( ns + "HintPath" ) )
				{
					tryAddAssemblyToCopy( Path.Combine( projectDirectory, (string)h ) );
				}
			}

			foreach ( var f in assembliesReturned )
			{
				yield return f;
			}
		}
	}
Advertisements
Categories: C#, MSBuild

FixedWidthStreamReader, Conceived from Death

October 1, 2014 1 comment

The other day, I had the task of converting an function that calculates annuity factors from VBA to C#.  A couple of the function parameters were the desired Male and Female mortality tables.  Thus the pun with the post title (…From Death).

tl;dr – Jump to the code to see how I used Generics, Compiled Expression Trees, and Convert.ChangeType to create my FixedWidthStreamReader.

Given the desired tables, the old VBA code would read in all the rates from text files…120 rows and 2-3 columns in fixed width format.  Not being expert in file streams, I hit Google and ended up at the Stack Exchange post How to read fixed-width data fields in .Net.  You can read the post and the comments, but essentially, the original post required caller to call the following method for each ‘property’ they wanted read.  Additionally, you had to call the properties in the correct order as they appeared on the line since you didn’t pass a position in, but just a size that the reader internally used to keep track of the position.

T Read<T>( int size );

One of the comments suggested mimicking the StructLayoutAttribute which made sense as it keeps the file layout definition neatly packed away as attribute decorations on the output class instead of requiring each client of the reader class to know this definition.

The original poster never changed his code to use attributes and then I received an email from another user who commented on that post (Thanks Giuseppe) and posted his own implementation on the Stack Overflow post Reading data from Fixed Length File into class objects.  Given that inspiration, I decided to update the code with the given goals:

  1. The read<T>( T data ) method immediately jumped out to me as something that I thought I could simplify using Convert.ChangeType.
  2. I wanted to be able to read all the lines of a file and return a list of desired T objects. 
  3. It seemed that if you wanted to read more than one line in the original SO solution, that you would have to manage the parent Stream independently of the FixedLengthReader stream.
  4. I didn’t want to have to have the caller have to instantiate a T object and pass it into the reader.  I wanted the reader to return null or T objects appropriately.

So below is my attempt at a solution.  I derived from StreamReader, but since I really only intend the caller to use ReadLine or ReadAllLines, it may be better to change to just have a private StreamReader inside class.

Additionally, I’m no expert at streams, so not sure how much more performant the Stack Exchange solution’s ReadToBuffer is versus just a simple ReadLine, but here is my code.

FixedWidthStreamReader Code Snippets

Given file of: 
Person 1          1973 100000.54 
Grandparent 1     1950 2000000 

 

This is my type T class that each row in a file represents.  Each field that I want read in is decorated with a Layout attribute describing the start position and the length>

class PersonalInfo
{
   [Layout(0, 20)]
   public string Name;
   [Layout(20, 5)]
   public int YOB;
   [Layout(24, 15)]
   public double Pay;

   public override String ToString() {
       return String.Format("String: {0}; int: {1}; double: {2:c}", Name, YOB, Pay);
   }
}

LayoutAttribute class (note, I should probably support Properties in addition to Fields, but that would be an easy fix).

[AttributeUsage(AttributeTargets.Field)]
class LayoutAttribute : Attribute
{
   public int Index { get; private set; }
   public int Length { get; private set; }

   public LayoutAttribute( int index, int length )
   {
       Index = index;
       Length = length;
   }
}

I will paste FixedWidthStreamReader code below, but here are three usage code scenarios.

1.  Read just the first line of file into a PersonalInfo object.

using( var sr = new FixedWidthStreamReader<PersonalInfo>( @"c:\users\terry.aney\desktop\fixed.txt" ) )
{
	var pi = sr.ReadLine();
	System.Diagnostics.Debug.WriteLine( pi.ToString() );
}

2. Read all lines of file into IEnumerable<PersonalInfo> object.

using( var sr = new FixedWidthStreamReader<PersonalInfo>( @"c:\users\terry.aney\desktop\fixed.txt" ) )
{
	foreach( var pi in sr.ReadAllLines() )
	{
		System.Diagnostics.Debug.WriteLine( pi.ToString() );
	}
}

 

3. Read all lines of a file one at a time into PersonalInfo objects (might want to do this if you want to stop reading when reach someone or something).

using( var sr = new FixedWidthStreamReader<PersonalInfo>( @"c:\users\terry.aney\desktop\fixed.txt" ) )
{
	PersonalInfo pi;
	while( ( pi = sr.ReadLine() ) != null )
	{
		System.Diagnostics.Debug.WriteLine( pi.ToString() );
	}
}


Finally, here is the FixedWidthStreamReader class.  I create compiled Lambda expressions for each Field on the object and call the compiled Action during ReadT()

public class FixedWidthStreamReader<T> : StreamReader where T : class
{
	private List<Tuple<int, int, Action<T, string>>> propertySetters = new List<Tuple<int, int, Action<T, string>>>();
	
	public FixedWidthStreamReader( Stream stream ) : base( stream ) { GetSetters(); }
	public FixedWidthStreamReader( string path ) : base( path ) { GetSetters(); }
	
	private void GetSetters() 
	{
		var myType = typeof( T );
		var instance = Expression.Parameter( myType );
		var value = Expression.Parameter( typeof( object ) );
		var changeType = typeof( Convert ).GetMethod( "ChangeType", new[] { typeof( object ), typeof( Type ) } );

		/* Should probably do Properties here too, if I change AttributeUsage for LayoutAttribute I would */
		foreach ( var fi in myType.GetFields() )
		{
			var la = fi.GetCustomAttribute<LayoutAttribute>();
			if ( la != null )
			{
				var convertedObject = Expression.Call( changeType, value, Expression.Constant( fi.FieldType ) );

				var setter = Expression.Lambda<Action<T, string>>(
					Expression.Assign( Expression.Field( instance, fi ), Expression.Convert( convertedObject, fi.FieldType ) ),
					instance, value
				);
				
				var prop = setter.Compile() as Action<T, string>;
				propertySetters.Add( Tuple.Create( la.Index, la.Length, prop ) );
			}
		}
	}

	public new T ReadLine()
	{
		if ( Peek() < 0 ) return (T)null;
		
		return ReadT( base.ReadLine() );
	}
	
	private T ReadT( string line )
	{
		if ( string.IsNullOrEmpty( line ) ) return null;
		
		var t = Activator.CreateInstance<T>();

		foreach( var s in propertySetters )
		{
			var l = line.Length;
			
			if ( l > s.Item1 )
			{
				s.Item3( t, line.Substring( s.Item1, Math.Min( s.Item2, l - s.Item1 ) ).Trim() );
			}
		}
		return t;
	}
	
	public IEnumerable<T> ReadAllLines()
	{
		string line = null;
		
		while ( !string.IsNullOrEmpty( ( line = base.ReadLine() ) ) )
		{
			yield return ReadT( line );   
		}
	}
}

 

Ultimately, I used this as a VERY SIMPLE learning experience on how to create compiled Expression Trees, which I still need to mess with more to fully understand, but hopefully this will give you both a simple introduction to that along with a pretty functional FixedWidthStreamReader.

Categories: C#, Expression Tree, Generics

Use SelectExcept When You Are Too Lazy To Type

February 27, 2013 Leave a comment

Do you love LINQPad?  Use it every day?  Do ad-hoc queries non-stop against your DataContexts?  Love C#/LINQ’s ability for anonymous object projections into new forms?  I’m assuming you said yes to all those.  Here is the question that drives this post…do you hate having to type *every* field *except* one (or a few) when you *only* need almost all fields?  If you answered yes to that last question or are wondering why I don’t just allow the entire row to be selected/returned, continue reading…

Read more…

Categories: C#, Extension Methods, LINQ

L2S Extensions…MVC style

February 9, 2012 1 comment

Four years later…I’ve finally come up for breath.  I hope to start a better pattern of trying to get some technical posts created once in a while.  Until I have my first inspiration – that I feel would be beneficial to anyone in the WWW, I thought I’d provide a little update to my L2S Extensions.

Here is what has changed since my last post 4 years ago.  I’m going to make the assumption that since there was really only three new improvements, that the code most have been pretty solid Smile.  Some of the changes below were due to the fact that the main code I generate for my company has shifted to the ASP.NET MVC framework and have started leveraging the System.ComponentModel.DataAnnotations namespace to decorate pretty much everything.  Since I was querying some of those same classes, it made sense to update my L2S Extensions code to obey some attributes as well.  All the changes below occur in DumpCSV().

Read more…

Categories: C#, Extension Methods, LINQ

I’ve Left Query Analyzer Hell For LINQPad Heaven

December 4, 2008 13 comments

So now that LINQPad has enabled intellisense SQL Server Management Studio, Query Analyzer, and even Joseph Albahari’s (LINQPad creator) own QueryEx have all been zapped from my memory.  I’ll no longer flounder in antiquated ANSI SQL, but instead flourish in fully typed C#/LINQ code.

But wait, there’s more, click to keep reading…

Categories: C#, Extension Methods, LINQ

LINQ to SQL Batch Updates/Deletes: Fix for ‘Could not translate expression’

April 20, 2008 14 comments

I’ve found and posted a new fix in the code from my original post: Batch Updates and Deletes with LINQ to SQL.  I’m not sure of the etiquette for this sort of thing: new post (like I’m doing) or just a comment in the original post.  But since I did get a fair amount of hits to the article but minimal comments, people who may have downloaded the code wouldn’t get an update notification and I want to be sure to make them aware of an issue/fix (assuming they are monitoring via a RSS feed).

But wait, there’s more, click to keep reading…

Categories: C#, LINQ

Batch Updates and Deletes with LINQ to SQL

April 14, 2008 47 comments

A couple weeks ago, I read the article, LINQ to SQL Extension: Batch Deletion with Lambda Expression by Jeffrey Zhao.  In case you didn’t read the article, it discusses the downside of most O/R Mapping frameworks when it comes to multiple updates or deletes.  He states the fact that a SQL statement for each row flagged as update/delete in the entity set is created.  I went about implementing something similar to what Jeffrey envisioned and I’ll explain some of the hurdles I had to overcome to achieve it.

But wait, there’s more, click to keep reading…

Categories: C#, Extension Methods, LINQ