Custom Component vs Script Component

RezaLeave a comment

Why this comparison?

Recently while working on a (relatively) huge cloud migration project, I had the opportunity to brush up on my SSIS knowledge that was getting ancient. I had to work on several SSIS projects and pave the way for my fellow developers so we could all move faster.

When working on SSIS projects, often times there is need to write logic that cannot be achieved by using out-of-the-box components from the toolbox. In those cases the first option that automatically comes to mind is Script Component. This component lets you write code in your language of choice to act as a source, destination or transformation. You always also have the option of making your own shiny custom component. It is generally believed that making a custom component is usually more complicated and it takes more time so most of the times developers use Script Component throughout the project and it leads to duplicating code that is not maintainable and every change is hard to track to a point that in larger projects no one is really sure how each part works anymore 🙂

What’s wrong with Script Component

Script Component looks pretty good in theory. It gives you the possibility to embed code in a data flow. It can act as a source, destination or a transformation. The problem is in the way the code is embedded in the package. If you open the XML content of a package that has a Script Component is it, you will notice that it is actually a full Visual Studio project with a random generated namespace and some generated code to make it easy to interact with fields through typed properties. When you double click on a Script Component, SSDT decodes that projects and (presumably) writes all the files of that project in a temporary folder and opens it in a new instance of Visual Studio. When you close that project after applying your changes, it has do do the reverse. If you put a break-point somewhere in that project, then again it has to do all that again. So, to sum it up:

  • Script Components take large space in SSIS packages and if you create several Script Components in a package, it can dramatically increase the size of the package hence slowing down development and stability of Visual Studio and hinder productivity
  • Each Script Component is an independent project that makes debugging them harder specially that they are encoded and embedded within the package’s XML.
  • The fact that all Script Components are encoded and embedded within the XML file will make it really hard to do SSIS development in a team and it is very difficult to trace a change no matter which source control software you use. Imagine comparing thousands of lines of encoded XML to see where some logic might have been changed in some lines of scripts.
  • The debugger in Visual Studio might not work properly (even crash) when you want to debug multiple Script Components. It is also not possible to have breakpoints in more than one Script Component
  • Readability is another big issue. You will not be able to easily skim through your Script Components to understand the logic. Imagine you will need to open each one by one slowly and if you miss something, you will need to back again to a previous one. It is not like a typical Visual Studio project where you can easily navigate through code with several out-of-the-box features.
  • The name of the columns you get in the Script Component might not be exactly the same as original columns, if you decide to use generated typed properties.
  • It is not possible to reuse the logic that is inside a script component, unless if you externalize it in another assembly and in that case managing versions of that external assembly can become tedious, because you might need to open every single Script Component that references that assembly should the version change and if you by mistake do a breaking change you will not find out during compile time. You will need to test every single script component using that assembly and there is no easy way to find out which Script is referencing that. so, in short you will need a full end to end test to find an issue that you could be found in a compile in other types of projects.
  • It is not easy to duplicate a Script Component. I know that it is against DRY rule to duplicate code, but lets say due to above limitation in special scenario you decide to do so, in that case because the assembly name in those Script Components will be the same, during compile time one of will replace the other without SSDT giving you an error or even a warning.
  • This one is actually a bug. If you use more recent versions of C# language (I don’t know for VB, but probably the same) none of your break point will ever be hit. So, it is better to forget about those shiny features that came with C# 6.0+ like simplified dictionary initializer (i.e. { ["key"] = value }).

What can justify using a Script Component?

In my opinion with all the above problems around Script Components, the only reason you should use it is when you need to create a one off component as part of your data flow or when your are going to have just a few minimal Script Components in your package. In all other cases that I have experienced it is better to create a custom component. You might be thinking that building a custom component can be difficult or you might have already tried to find your way around MSDN to figure out how to do it, but got confused with the minimal explanation and over simplified examples that exists. But, trust me on this, it is not that difficult if you do it right. In my next post, I’m going to show you just that, but with a clear step by step guide and a real world example.

Custom Component vs Script Component

Development behind corporate proxy + authentication

Reza, , , , , , Leave a comment

Developing in companies that have proxy servers for developers can be frustrating in this age when every tool needs access to online resources and even parts of software development life cycle are cloud based. Proxy servers that require NTLM authentication just add to that frustration. NTLM is developed by Microsoft but many applications built by Microsoft do not support it or require some configuration and in worst cases some hacking to make it work. Below is a list of some the tools that developers might be using on a daily basis and what you need to do to make them connect via NTLM proxy. I keep adding more to the list as I encounter them.

  • Visual Studio Code (VSCode)
  • NPM
  • Visual Studio, Web Platform Installer and other .NET Applications

Visual Studio Code (VSCode)

VSCode 1.15 and up now supports NTLM proxy (finally Microsoft supported its own authentication protocol).

NPM

For NPM you have two options. Either to send proxy address and credentials in every single command you run or to set them in the global configuration of NPM. I recommend the former because it is more secure.

Set proxy in every command

When calling NPM command you can always use --proxy switch to set proxy for each command. The syntax for using this switch is the following.

--proxy username:password@proxyaddress:port

For example to use myproxy:8080 as proxy address and my-domain\reza as username and P@ssw0rd as password when calling the install command you can type the following.

npm install --proxy http://my-domain%5Creza:P%40ssw0rd@myproxy:8080

Please note that both username and password are URL encoded. You can use the following command in your browser’s developer tools to encode them.

encodeURI('my-domain\\reza')
encodeURI('P@ssw0rd')

Set proxy in NPM configuration

To set the proxy in the global configuration of NPM you need use the same format as above for sending the proxy address, username and password and use npm config set to store it in the configuration. For example to set the proxy address to myproxy:8080 and username to my-domain\reza and password to P@ssw0rd you can use the following command.

npm config set proxy http://my-domain%5Creza:P%40ssword@myproxy:8080

Visual Studio, Web Platform Installer and other .NET applications

To set the proxy for pretty much any .NET application, you need to put the following in the configuration file of that application.

<system.net>
    <defaultProxy useDefaultCredentials="true" enabled="true">
        <proxy bypassonlocal="true" proxyaddress="http://myproxy:8080" />
    </defaultProxy>
</system.net>

For Visual Studio I suggest also enabling IPV6 if the above configuration did not work as suggested by some other developers.

<system.net>
    <settings>
        <ipv6 enabled="true"/>
    </settings>
    <defaultProxy useDefaultCredentials="true" enabled="true">
        <proxy bypassonlocal="true" proxyaddress="http://myproxy:8080" />
    </defaultProxy>
</system.net>

For executable files the configuration file is named the same as executable’s file name but with .exe.config extension. For Visual Studio it is called devenv.exe.config and for Web Platform Installer it is WebPlatformInstaller.exe.config.

Postman

At the moment Postman (v7.36.1) does not support NTLM authentication and the only best way that you can make it work is by using Fiddler. Here is what you need to do, step by step:

  1. Install Fiddler and run it.
  2. Make sure Rules > Automatically Authenticate is selected. This will enable Fiddler to authenticate on behalf of Postman with your current user account.
  3. * In Postman, go to File > Settings and then Proxy and turn on Global Proxy Configuration
  4. For Proxy Type select both HTTP and HTTPS
  5. For the Proxy Server, use 127.0.0.1:8888 (If you you have changed the default port that Fiddler is listening on you will need to change it here as well).
  6. ** In Fiddler, go to Tools > Options... and then in HTTP tab, select Capture HTTPS CONNECTs and Decrypt HTTPS traffic and install the certificate when prompted.
  7. In Postman go to File > Settings > General, turn off SSLS certificate verification. You need to do this because currently, Postman does not support intermediate proxies.

* In step 6, instead of using Global Proxy Configuration you may also use Use System Proxy, but in that case you need to make sure in Fiddler, Capture Traffic is selected under File menu. This way Fiddler will capture all the HTTP traffic by setting Windows Proxy settings.

** You won’t need to do step 6 and 7 if you won’t be working with HTTPS URLs.

Development behind corporate proxy + authentication

Developing NodeJS apps using AngularJS 4 in macOS

RezaLeave a comment

Well, that is one long title for a blog post! but, bare with me I’m going straight to the point. Recently I’ve been wanting to dip my toes in the AngularJS 4 world and to make it more exciting I thought maybe I will do it on a Mac using NodeJS and VSCode. Now that VSCode is getting more and more popular sharing my experience here might actually help someone and a future me to have an easier ride than mine today.

The ingredients

  • A computer with macOS on it. Any version would do.
  • Homebrew
  • VSCode
  • Git client for Mac
  • NodeJS
  • Half an hour of time!

Preparation

We first need to get the environment ready before we start to develop something and believe it or not there are several ways you can do that. After studying and trying out different ways, this is the best way I found that is the safest, fastest, easiest, most repeatable and undoable, and yet inline with best practices of Apple in Mac world!

  1. Install Homebrew if you don’t already have it.
  2. Install NPM
  3. Install VSCode or any other code editor / IDE
  4. Install Git client
  5. Install NodeJS
  6. Make a folder
  7. Start VSCode

1. Install Homebrew

Although NodeJS website provides an installer for NPM,  I suggest using Homebrew to install NPM and any other open source package you might need in you Mac. NPM installer needs admin rights and it will put files in folders that only admin users have access to. You don’t want any 3rd party application tinkering with those areas do you?! plus it will be easier to cleanup afterwards should you need to remove NPM for some reason in future.

I recommend you visit Homebrew’s official website https://brew.sh to know more about it. At the time of writing this blog post, you just need to open Terminal and run the following command to execute Homebrew’s installation script. The script will explain what it does and then pauses for your confirmation.


ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

2. Install NPM

Now that you have installed Homebrew, installing NPM is as simple as typing the following command in the Terminal window. It will install the latest stable version.


brew install node

After the installation is done, you can type the following command to see which version is installed.


node -v

If you need to update Node and NPM in future you can use brew upgrade node and to uninstall them you can use brew uninstall node.

3. Install Visual Studio Code

This step is pretty easy. You just need to download and install Visual Studio Code, Microsoft’s lightweight code editor from its official website and install it like any other application. Here is the official download page: https://code.visualstudio.com/download

Directions

There are two ways you can start to write a new application by AngularJS. You can either manually set up the project by creating the required boilerplate files and writing configurations which can really help you understand what is going on behind the scene or you can use the Angular CLI to do it for you and this way you save some time. I’m going to explain both methods here and first I am going to start with the manual method. Instead of explaining the manual method, I’m going to show you how to use Angular CLI because that’s the way you’re most probably going to do it in future. Then I will explain all the files and configurations in such a way that you can even do it manually if you wish to.

[To be continued!]

 

 

Developing NodeJS apps using AngularJS 4 in macOS

How to mimic Pivot Table or Categories in Number 3+

Reza17 Comments

Perhaps you are already aware that Apple has cut many features from latest versions of Numbers and their other MS Office like products. One of the most popular features that has been removed from Numbers is called Categories. By activating Categories on a table you could aggregate data and summarize values see an overview of what matters to you. Recently I was working on a list of values and since the list was not bigger than what Numbers can handle and I didn’t need the speed and power of Microsoft Excel. I though I will give Numbers a try. I still like the fact that you have a free canvas in Numbers that allows you to put put many tables independent from each other in one page. I think it is the only advantage of Numbers to its alternatives. Otherwise with just a few dollars per months you could have access to the latest version of Microsoft Office regardless of your OS or even directly in your browser plus Microsoft gives you a terabyte of online storage! OK, let’s get back to work before I change your mind 😉

Let’s start with an example like the following.

A B
1 Category Amount
2 Blue 11.20
3 Red 15.89
4 Red 10.30
5 Orange 32.12
6 Green 15.39
7 Blue 10.18
8 Green 24.76
9 Green 89.31
10 Orange 8.75
11 Blue 15.28

The first column is just containing the order of each row, the second indicates the category of each row. This could be anything like category of expenses in an expense report. It is actually the column that we are going to aggregate. The third column contains the values. It could be the amount of each expense for example.

Now, let’s say you want to aggregate it to the following table. But you want Numbers to do it for you so each time a series of row is added to the table you don’t need to calculate everything manually.

Category Sum
Blue 36.66
Red 26.19
Orange 40.87
Green 129.46

Since I want to do the aggregation in the same table, I need to find a way to detect distinct categories. Then per each category I need to calculate and display the sum of each category in front of it. If I display only the first occurrence of each category and hide other occurrences the problem would be solved. Let’s split the problem to smaller pieces as usual. Now the first question would be how do we find distinct categories?

Step 1 – Detecting distinct categories

Let’s add a new column called “Test” then put the following formula in the first cell of the column below the header: =COUNTIF(A$1:A1,A2) in other words you need to put it in C2 cell. Now click at the bottom of the cell’s rectangle and drag it to the bottom of the table. This will fill remaining cells of the column. The result will be the following table.

A B C
1 Category Amount Test
2 Blue 11.20 0
3 Red 15.89 0
4 Red 10.30 1
5 Orange 32.12 0
6 Green 15.39 0
7 Blue 10.18 1
8 Green 24.76 1
9 Green 89.31 2
10 Orange 8.75 1
11 Blue 15.28 2

As you can see, you only get zeros for the first occurrence of each category and that means we can detect them with an IF. Now, let’s put the following formula in cell C2 instead: =IF(COUNTIF(A$1:A1,A2)=0,"*","") and as before drag the corner of the cell until the bottom of the table which will give us the following result.

A B C
1 Category Amount Test
2 Blue 11.20 *
3 Red 15.89 *
4 Red 10.30
5 Orange 32.12 *
6 Green 15.39 *
7 Blue 10.18
8 Green 24.76
9 Green 89.31
10 Orange 8.75
11 Blue 15.28

Now for first occurrence of each category we are displaying “*” in column C.It means that we are able to detect the first occurrence of each category and display an arbitrary text in it. What about displaying SUM of each category there. This would be the final solution.

My final solution

In this step we are going to replace the “*” in the formula with SUMIF(Category,C2,Amount) this way the SUM of each category will be calculate in front of each occurrence of that category. All you have to do is to put the following formula in C2, then drag the bottom-right corner of the cell until the bottom of the column.
=IF(COUNTIF(C$1:C1,C2)=0,SUMIF(Category,A2,Amount),””)
The resulting table will be similar to the following. I just renamed the Test column to “Total per category”. You can use formatting to distinguish the aggregate row or even hide other rows if you like.

A B C
1 Category Amount Total per category
2 Blue 11.20 36.66
3 Red 15.89 26.19
4 Red 10.30
5 Orange 32.12 40.87
6 Green 15.39 129.46
7 Blue 10.18
8 Green 24.76
9 Green 89.31
10 Orange 8.75
11 Blue 15.28
How to mimic Pivot Table or Categories in Number 3+

How to warm-up SharePoint or other web applications and WCF (SOAP) services with PowerShell

Reza, , , Leave a comment

There are many reasons you might want to warm-up a web application occasionally. It can be after a fresh deployment or on a regular basis after recycling application pools. Some times you might also need to warm-up SOAP services without going through front-end.

It might seems to be any easy task specially if you have PowerShell 3.0 or higher on your servers, but after Googling a while and reviewing some of the top hits I discovered that each solution is missing a part. Some only work in a single server scenario and some has forgotten that each HTTP response might contain links to scripts and images that we need to download and finally I could not find anything for SOAP services that just works. Long story short I decided to put together a simple script that just works and is easy to change to everyone’s needs.

Please note that my only assumption is you have PowerShell 3.0+ in your servers.

Currently the script takes care of the following tasks, but I will most likely improve it to cover other useful scenarios.

  • Calling SOAP operations and sending parameters and custom headers
  • Calling front-end URIs and downloading scripts and images that are local to the front-end
  • Logging to a configurable folder and file name
  • Cleaning up old log files

Currently, I have the following points in mind to improve the script.

  • Put configuration in a different file.
  • Improve function definitions (named and typed parameters and description).
  • Default values for parameters when it makes sense (e.g. log folder can be the current folder).
  • Support REST services.
#Parameters
$logFolder = "C:\Temp\logs"
$logFile = "$logFolder\Warmup_$(Get-Date –Format "MM-dd-yyyy_hh-mm-ss").log"
$timeoutSec = 10
$headers = @{
SOAPAction= "";
"Accept-Encoding" = "gzip,deflate";
"User-Agent" = "PowerShell/Warmup-WebApp"
}
$webRequests = @(
@{
SOAPAction = "http://tempuri.org/IService1/Operation1";
Body = @"
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/&quot; xmlns:tem="http://tempuri.org/"&gt;
<soapenv:Body>
<tem:Operation1>
<tem:Parameter1>1</tem:Parameter1>
<tem:Parameter2>2</tem:Parameter2>
</tem:Operation1>
</soapenv:Body>
</soapenv:Envelope>
"@
},
@{
SOAPAction = "http://tempuri.org/IService1/Operation2";
Body = @"
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/&quot; xmlns:tem="http://tempuri.org/"&gt;
<soapenv:Body>
<tem:Operation1>
<tem:Parameter1>1</tem:Parameter1>
<tem:Parameter2>2</tem:Parameter2>
</tem:Operation1>
</soapenv:Body>
</soapenv:Envelope>
"@
}
)
# Functions
function Write-Log([string]$level, [string]$text) {
$logTime = Get-Date –Format "yyyy-MM-dd HH:mm:ss.fff"
Write-Host "[$level] $text"
if ($logFile -ne $null) {
"$logTime [$level] $text" | Out-File –FilePath $Logfile –Append
}
}
function Cleanup-LogFiles() {
$expiryDate = (Get-Date).AddMinutes(-7)
[bool]$logFileDeleted = $false
$filesToDelete = Get-Childitem $logFolder –Include "*.log" –Recurse | ? {$_.LastWriteTime -le $expiryDate} | % {
Remove-Item $_.FullName | out-null
Write-Log "Information" "File deleted successfully – $_"
$logFileDeleted = $true
}
if (-not $logFileDeleted) {
Write-Log "Information" "No old log file was found in $logFolder"
}
}
function Warmup-WcfService($serviceUrl) {
foreach($webRequest in $webRequests) {
$headers["SOAPAction"] = $webRequest.SOAPAction
Invoke-WcfService $serviceUrl $headers $webRequest.Body
}
}
function Invoke-WcfService($uri, $headers, $body) {
$invocationStartTime = Get-Date
try {
$result = Invoke-WebRequest –Uri $uri –Method Post –UseDefaultCredentials –ContentType "text/xml; charset=utf-8" –Headers $headers –body $body –TimeoutSec $timeoutSec
$invocationDuration = (Get-Date).Subtract($invocationStartTime)
if ($result.StatusCode -eq 200) {
if ($result.Headers.ContainsKey("Content-Length")) {
$contentLength = $result.Headers["Content-Length"]
} else {
$contentLength = "$($result.RawContentLength) (raw)"
}
Write-Log "Information" "WCF operation invoked successfully – Operation: $($webRequest.SOAPAction)($variation), Duration: $($invocationDuration), Length: $contentLength"
} else {
Write-Log "Warning" "WCF operation failed – Operation: $($webRequest.SOAPAction)($variation), Status: $($result.StatusCode) $($result.StatusDescription), Length: $contentLength"
}
} catch {
Write-Log "Critical" "WCF operation failed – Operation: $($webRequest.SOAPAction)($variation), Exception: $_"
}
}
function Invoke-WebPage([string]$uri, [bool]$getImages, [bool]$getScripts) {
try {
$response = Invoke-WebRequest –Uri $uri –UseDefaultCredentials –TimeoutSec $timeoutSec –Proxy "http://localhost" –ProxyUseDefaultCredentials
if ($response.StatusCode -eq 200) {
Write-Log "Information" "Web page invoked successfully – URL: $uri, Images: $($response.Images.Count), Scripts: $($response.Scripts.Count), Links: $($response.Links.Count)"
} else {
Write-Log "Warning" "Web page invocation failed – URL: $uri, Status: $($response.StatusCode) $($response.StatusDescription)"
}
} catch {
Write-Log "Critical" "Web page invocation failed – URL: $uri, Exception: $_"
}
if ($getImages) {
Fetch–Resources $uri ($response.Images | ? { $_.src } | % { $_.src })
}
if ($getScripts) {
Fetch–Resources $uri ($response.Scripts | ? { $_.src } | % { $_.src })
}
}
Function Fetch-Resources($baseUrl, $resources) {
[uri]$uri = $baseUrl
$rootUrl = $uri.Scheme + "://" + $uri.Authority
$counter = 0
foreach ($resUrl in $resources) {
if ($resUrl.StartsWith("http", [System.StringComparison]::OrdinalIgnoreCase)) {
$fetchUrl = $resUrl
} elseif (!$resUrl.StartsWith("/")) {
$fetchUrl = $rootUrl + "/" + $resUrl
} else {
$fetchUrl = $rootUrl + $resUrl
}
# Progress
Write-Progress –Activity "Opening " –Status $fetchUrl –PercentComplete (($counter/$resources.Length)*100)
$counter++
# Execute
try {
$resp = Invoke-WebRequest –UseDefaultCredentials –UseBasicParsing –Uri $fetchUrl –TimeoutSec 120
Write-Log "Information" "Resource has been downloaded successfully – URL: $fetchUrl"
} catch {
Write-Log "Warning" "Resource failed to download – URL: $fetchUrl, Exception: $_"
}
}
Write-Progress –Activity "Completed" –Completed
}
# Execution
Write-Log "Information" "My Web Application Warmup Script – $(Get-Date)"
Write-Log "Information" "Cleaning up old log files…"
Cleanup–LogFiles
Write-Log "Information" "Warming up WCF Services…"
Warmup–WcfServices
Write-Log "Information" "Warming up Front-End…"
Invoke-WebPage "http://domain.com/area1/Page1.aspx" $true $true
Invoke-WebPage "http://domain.com/area1/Page2.aspx" $true $true
Write-Log "Information" "My Web Application Warmup Script Completed."

view raw
Warmup-WebApp.ps1
hosted with ❤ by GitHub

I’m open to any suggestion and feature request. Please let me know if you found it useful or if have got something wrong.

How to warm-up SharePoint or other web applications and WCF (SOAP) services with PowerShell