3/18/2022

ASP.NET Core Middleware to Update Response Body

I needed a quick way to update a mix of static HTML files and Razor views to change a message common to a lot of pages. In the example here I will update the Copyright message in HTML pages.

As usual I searched the web and found many examples, none that worked the way I needed (or worked at all!), but I did find an assortment of "bits" scattered throughout StackOverflow that helped me build a working solution.

At first glance this should have been a textbook example of ASP.NET Core middleware. Intercept the outgoing response, update the body HTML and forward on through the middleware pipeline. The first issue was the Stream object used by Response.Body object, its an Microsoft.WebTools.BrowserLink.Net.ScriptInjectionFilterStream object that does not support Seek. The first "bit" that I found in StackOverflow was to replace the Response.Body steam with my own stream, in particular a MemoryStream. Then we can send the Response object on down the middleware pipeline using next.Invoke. When the Response comes back to us though the pipeline we need to extract the HTML text from our custom stream object, do our updates to the HTML and finally get the original kind of stream back into the Response.Body with our changes.

The next problem was that I could not find a way to clear or reset the stream that represented the returned Body content. I was getting both the original page plus the updated page, i.e. my data was getting appended. .SetLength(0) and .Seek(0, SeekOrigin.Begin) did not work. ("Specified method is not supported"). StackOverflow to the rescue again. A quick solution was to recreate an empty stream object. ( context.Response.Body = new MemoryStream(); )

Remember! The order of middleware is important. If you want this to process your JavaScript, CSS and static HTML files, then add this before app.UseStaticFiles(). Most likely order:

            app.UseHttpsRedirection();
            app.UseStaticFiles();
            app.Use(  the code below  );
            app.UseRouting();
            app.UseAuthorization();

 

So here's the solution as an inline "app.Use()" block added to Startup.cs or Program.cs (.NET 6). It replaces the Copyright date from "2021" to the current year.

 

Note: Code tested in .NET Core 3.1 and .NET 6.0 using Visual Studio 2022.


app.Use(async (context, next) =>
{
    using (var replacementStream = new MemoryStream())
    {
        // preprocessing the Body
        // replace {Microsoft.WebTools.BrowserLink.Net.ScriptInjectionFilterStream}
        // with a MemoryStream (seekable!)
        System.IO.Stream originalStream = context.Response.Body;
        context.Response.Body = replacementStream;

        // do any preprocessing of the Request or Response object here
        // none needed for this example

        // move on down the pipeline, but with our custom stream
        await next.Invoke();

        // we are now back from the rest of the pipeline and on the way out (TTN)

        // postprocessing the Body

        // read the returned Body
        replacementStream.Seek(0, SeekOrigin.Begin);
        using (var bufferReader = new StreamReader(replacementStream))
        {
            string body = await bufferReader.ReadToEndAsync();

            // Is this a web page?
            if (context.Response.ContentType.Contains("text/html"))
            {
                // make changes to the returned HTML
                // set the copyright to the current year
                body = body.Replace("© 2021", "© " + DateTime.Now.Year.ToString());
            }

            // discard anything in the existing body.                        
            context.Response.Body = new MemoryStream();

            // write the updated Body into the Response using our custom stream
            context.Response.Body.Seek(0, SeekOrigin.Begin);
            await context.Response.WriteAsync(body);

            // extract the Body stream into the original steam object
            //   in effect, covert from MemoryStream to 
            //   {Microsoft.WebTools.BrowserLink.Net.ScriptInjectionFilterStream}
            context.Response.Body.Position = 0;
            await context.Response.Body.CopyToAsync(originalStream);

            // now send put the original stream back into the Response
            // and exit to the rest of the middleware pipeline
            context.Response.Body = originalStream;
        }
    }
});


 

Here's the same solution as a Middleware class.

  1. Add the class below.
  2. Update Startup.cs or Program.cs (.NET 6) to add an app.UseReplaceText line.

 Startup.cs or Program.cs:

app.UseHttpsRedirection();
app.UseStaticFiles();
app.UseReplaceText();  // custom middleware
app.UseRouting();
app.UseAuthorization();
app.UseEndpoints(endpoints => ....

The class:

using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Http;
using System;
using System.IO;
using System.Threading.Tasks;

namespace WebAppToDemoMiddleware
{
    public class ReplaceTextMiddleware
    {
        private readonly RequestDelegate _next;

        public ReplaceTextMiddleware(RequestDelegate next)
        {
            _next = next;
        }
        // 
        public async Task InvokeAsync(HttpContext context)
        {
            using (var replacementStream = new MemoryStream())
            {
                // preprocessing the Body
                // replace {Microsoft.WebTools.BrowserLink.Net.ScriptInjectionFilterStream}
                // with a MemoryStream (seekable!)
                System.IO.Stream originalStream = context.Response.Body;
                context.Response.Body = replacementStream;

                // do any preprocessing of the Request or Response object here
                // none needed for this example

                // move on down the pipeline, but with our custom stream
                await _next(context);

                // we are now back from the rest of the pipeline and on the way out

                // postprocessing the Body

                // read the returned Body
                replacementStream.Seek(0, SeekOrigin.Begin);
                using (var replacementStreamReader = new StreamReader(replacementStream))
                {
                    string body = await replacementStreamReader.ReadToEndAsync();

                    //if (body.Contains("Welcome"))
                    // Is this a web page?
                    if (context.Response.ContentType.Contains("text/html"))
                    {
                        // make changes to the returned HTML
                        // set the copyright to the current year
                        body = body.Replace("© 2021", "© " + DateTime.Now.Year.ToString());
                    }

                    // discard anything in the existing body.                        
                    context.Response.Body = new MemoryStream();

                    // write the updated Body into the Response using our custom stream
                    context.Response.Body.Seek(0, SeekOrigin.Begin);
                    await context.Response.WriteAsync(body);

                    // extract the Body stream into the original steam object
                    //   in effect, covert from MemoryStream to 
                    //   {Microsoft.WebTools.BrowserLink.Net.ScriptInjectionFilterStream}
                    context.Response.Body.Position = 0;
                    await context.Response.Body.CopyToAsync(originalStream);

                    // now send put the original stream back into the Response
                    // and exit to the rest of the middleware pipeline
                    context.Response.Body = originalStream;
                }
            }

        }
    }

    // Add an extension class so we can use this as app.UseReplaceText()
    public static class RequestCultureMiddlewareExtensions
    {
        public static IApplicationBuilder UseReplaceText(
            this IApplicationBuilder builder)
        {
            return builder.UseMiddleware<ReplaceTextMiddleware>();
        }
    }
}
..

7/09/2021

Power BI Books

I am often asked for book recommendations in my classes. Book choices are a personal thing that varies based on your learning style. So, I rarely recommend any particular book. The books that follow are books that I really like and recommend.


Storytelling with Data: A Data Visualization Guide for Business Professionals

This is not a Power BI book. Before you create a report, you should know why you are creating the report... the story you want to tell. Which visuals should you use, what is clutter and what is fact, which order should you list the data, all of those things you should know before you start your report project. This book covers all of that.

I bought it. I use it. I recommend it in every Power BI class.


Supercharge Power BI: Power BI is Better When You Learn To Write DAX

This is not a Power BI book, it is a book about using DAX formulas within Power BI. You cannot master Power BI without learning DAX. This book is written by a trainer in a tutorial and conversational format. It is full of examples and has a downloadable Excel version of the database for all of the examples.

The book is now in the third edition and was updated in April of 2021.

I got the book for free directly from Matt as a fellow MVP and I really like it!


8/04/2020

Merging JSON files using PowerShell


Let's say you have a directory full of JSON files and you need to merge them into a single file.


Emp1.json

    {
        "Name":  "Mike",
        "ID":  123,
        "Status":  "Active"
    }

Emp2.json

    {
        "Name":  "Susan",
        "ID":  124,
        "Status":  "Active"
    }

Emp3.json

    {
        "Name":  "Sam",
        "ID":  125,
        "Status":  "Inactive"
    }

You need a result that includes commas and square brackets that looks like this:

[
    {
        "Name":  "Mike",
        "ID":  123,
        "Status":  "Active"
    },
    {
        "Name":  "Susan",
        "ID":  124,
        "Status":  "Active"
    },
    {
        "Name":  "Sam",
        "ID":  125,
        "Status":  "Inactive"
    }
]



Solution 1:

Combing the files using Get-Content and append the start and end brackets.

"[" + ((Get-Content Emp*.json -raw) -join ","  ) + "]" | Out-File AllEmp.json

Note that the "-raw" reads the entire file as a single string, not an array of strings. That's how we can add a comma between each block of JSON. So, each file is read as a single string, each string is added by Get-Content to an array. The array of JSON strings is then joined into a single long string with a comma between each JSON string.

You might want the brackets on their own lines, so add some line breaks.

"[`n" + ((Get-Content Emp*.json -raw) -join ","  ) + "`n]" | Out-File AllEmp.json

What if you wanted to sort them or manipulate the objects' properties as they are merged? You will need to convert them to objects first, then sort or manipulate the objects, and then convert them back to JSON.

( "[" + ((Get-Content Emp*.json -raw) -join ","  ) + "]"  | ConvertFrom-Json ) | sort name | ConvertTo-Json | Out-File AllEmp.json


What if you have a lot of files? The examples above bring all of the files into memory and create one large string. Here's a solution that reads each file and appends to the merged file.
  • The first line creates the new file and writes out the starting bracket.
  • The second line creates our delimiter and sets it "" for the first item.
  • The third line gets each JSON file and writes it to the new file with a leading delimiter. After the first file it sets the delimiter to ",".
  • The last line adds the closing square bracket.

"[" | Out-File AllEmps.json -Force
$delim = "";
Get-ChildItem Emp*.json | foreach { $delim + (Get-Content $_ -raw); $delim="," } | Out-File AllEmps.json -Append }
"]" | Out-File AllEmps.json -Append


8/03/2020

How to count lines of code using PowerShell

I've got a project... scattered through a number of folders are a bunch of C# files. How many lines of code do I have? (Bragging rights you know...)

You could open the project in Visual Studio and use the Analyze, Calculate Code Metrics. But that requires Visual Studio, and does not give me all the options I want. (But in many ways is better! Lines by file and by method are available there.)

I wanted to do this on raw files using PowerShell, so assuming your project is in C:\myProject and you only wanted to count ".cs" files...

  dir C:\myProject -Recurse *.cs | Get-Content | measure

Get-Content returns the lines from the file as an array, and Measure-Object counts things.

Want to count characters too?

  dir C:\myProject -Recurse *.cs |
    Get-Content |
      measure
-Line -Character

Your boss won't let you take credit for blank lines?

  dir C:\myProject -Recurse *.cs |
    Get-Content |
      where { $_.trim() -ne "" } |
        measure
-Line -Character

Your boss won't let you take credit for blank lines or comments? (I would pay extra for good comments!)

  dir C:\myProject -Recurse *.cs |
    Get-Content |
      where { $_.trim() -ne "" -and $_.trim() -notlike "//*" } |
        measure
-Line -Character

    (Sorry, but in this quick little exercise I did not deal with /* and */ block comments, only // comments.)

Want to include multiple file types?

  dir C:\myProject -Recurse -Include *.cs, *.cshtml |
    Get-Content |
      where { $_.trim() -ne "" -and $_.trim() -notlike "//*" } |
        measure
-Line -Character


Boy I've been working hard... and using wizards to write code... 😉

   Lines Words Characters Property
   ----- ----- ---------- --------
    2563           117084

4/25/2020

PowerShell: When does zero equal one? (Length property of a Directory)


You learn the most interesting things when teaching a class, especially when an off the cuff demo goes wrong.

During a simple demo of a calculated or derived column I did this:

For files, the output was as expected:


But for directories it returned this:


While the FileInfo object does have a Length property, the DirectoryInfo object does not. So what should we expect when both files and directories are in the same pipeline? Directories usually display a blank column for Length. Is this a Null? Is this an empty or one space string? Or is it just skipped because DirectoryInfo objects don't have a Length property?

Noting that 9.53674E-07 is 1 / 1MB, PowerShell is returning a 1 for the missing Length property, and not a zero or null as I was expecting.

Turns out that Length is both a property of FileInfo objects, and also a property of all PowerShell objects. In my example, DirectoryInfo objects did not have a Length property so $_.Length returns the underlying object's Length property.

Here's an example of a new "Object". Note that PowerShell thinks it has a length of 1 even though Length is not one of its properties. If it's not a defined property, where does it come from? PowerShell seems to treat all objects as collections, even if it is a single item. Collections have both a Count and a Length property. So, $x has a Length (and a Count) of 1.


There's a hint in the PowerShell help files about this. https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_properties?view=powershell-7  (or in PowerShell help about_Properties)



And, as the Help file says, it was different in PowerShell 2.0. If I launch PowerShell with the -Version 2.0 option, I actually get the result I expected in the original demo!





.

4/01/2020

SharePoint vs. Teams vs. OneDrive



A few notes from a recent discussion about where to store files:

 
  • SharePoint Everywhere:
    • SharePoint is behind OneDrive, SharePoint and Teams. Each exposes a different level of functionality.
    • In Teams you can display as “Teams Files”, or click a link and open the backing SharePoint site. In OneDrive, most SharePoint functionality is hidden.
    • SharePoint sites can have multiple libraries (similar to multiple drive letters/network shares) while OneDrive and Teams only expose a single library.
  • Content Location and Ownership:
    • OneDrive is associated with individual users, and the content is “owned” by individuals. (OneDrive is actually SharePoint, but most of the SharePoint functionally is hidden.)
    • SharePoint libraries are associated with a SharePoint Site and the content is “owned” by the Site Owners, who are delegated to that role and can be quickly replaced in the future.
    • Teams file storage is in a disguised SharePoint library. It is “owned” by the Team owners.
  • Content Sharing:
    • OneDrive “sharing” is somewhat “ad hoc” and not driven by corporate policy or best practices. OneDrive users can share with anyone.
    • SharePoint “sharing” can be “ad hoc”, but is usually managed by assigning permissions to users, much like network shares. If done right, SharePoint allows quick and easy auditing of “who has what access to this file”.
    • Teams content by default is shared with the team.
  • Content Sync
    • The default use of OneDrive is via the “Sync” feature. Multiple users editing/deleting sync’d content impacts all users syncing that content.
    • The default use of SharePoint and Teams is closer to network shares. Files are stored there, and downloaded, or viewed/edited online, as needed. Libraries can be mapped to local drive letters. We typically discourage any SharePoint library syncing.


Governance is very important, especially with OneDrive. Some things to consider:
  • Who “owns” a file? Storing the file in OneDrive implies that the user owns it. Storing it in Teams or SharePoint implies the organization owns it. If in OneDrive, what’s the impact if a user leaves the company or the department? If in Teams, what’s the impact if a Team is deleted?
  • Who should grant access to corporate content? A OneDrive user can share with anyone. It’s so easy, the user rarely is thinking about security. In SharePoint or Teams, an “owner” grants and removes permissions for users and can easily update and audit user access.
  • Which copy of a document is the “single source of truth”? If we each have a copy in our OneDrive or local drive, which one is the official version?
  • If data with a legal impact is stored, who should manage it? Each user with their OneDrive, a “content steward” who manages a SharePoint site? A Teams owner (who has been property trained on your governance)?



  •  

Note to spammers!

Spammers, don't waste your time... all posts are moderated. If your comment includes unrelated links, is advertising, or just pure spam, it will never be seen.