How to fix proxy authentication issues in your development environment

I have written about proxy authentication before and although this annoying problem has been around, many tools still don’t support it well. Specially when it comes to NTLM flavor of it that has been developed by Microsoft long time ago and even many Microsoft tools did not support it until recently.

Proxy servers in my opinion are not the best tools to control internet access. Security benefit they bring us are not much compared to productivity they take away from developers.

In the past I have used many different techniques to overcome this issue (e.g. CNTLM, custom local proxy and more) but the best way that almost always works reliably and securely is to run Fiddler with its “Capture Traffic” turned off (unless you need it for other reasons) and point from your blocked app to it. This way:

  1. Fiddler won’t be capturing all the traffic and it will only proxy the traffic for the applications that are specifically pointing to it.
  2. It takes care of authentication on behalf of you without worrying to store your credentials somewhere that is not safe or having to update your credentials when they change.
  3. You can always close Fiddler when you want to cut the traffic

In the rest of this blog post I will show how you can configure every development tool that I know (and remember) to point to Fiddler as a proxy.

Keep in mind that it doesn’t have to be Fiddler, any other local proxy that can take care of authentication on behalf of the logged in user can do the same.

Please let me know if I have forgotten a developer tool or if you know how to configure another tool and what others to know it. I will add it to the list as soon as possible.

Nuget Package Manager

At this moment Nuget Package Manager does not understand proxy authentication and you need to change its configuration to use your local proxy server (Fiddler) to take care of authentication. The configuration file is located in

%USERPROFILE%\AppData\Roaming\NuGet\NuGet.Config

<configuration>
  <config>
    <add key="http_proxy" value="http://localhost:8888" />
  </config>
</configuration>

Node Package Manager (NPM)

At this moment Nuget Package Manager does not understand proxy authentication and you need to change its configuration to use your local proxy server (Fiddler) to take care of authentication.Run the following command to know if your NPM is using any proxy.

Run the following command to know your current proxy setting (if any).

npm config get proxy

To set Fiddler as your proxy setting use the following command.

npm config set proxy http://localhost:8888

You have the possibility to point to your corporate proxy server using the following syntax.

npm config set proxy http://{domain\username-url-encoded}:{password-url-encoded}@{proxy-domain-name}:{port}

But keep in mind that if you directly point to your corporate proxy, every time your password changes you need to run the above command again to update the setting.

Side: To URL encode your username or password you can use encodeURIComponent in your browser’s Console. For example:

>> encodeURIComponent('domain\\username')
"domain%5Cusername"
>> encodeURIComponent("P@ssw0rd!")
"P%40ssw0rd!"

Visual Studio’s Task Runner (NPM in Visual Studio)

Whatever I explained about configuring NPM’s proxy setting previously is still applicable here, but you need to keep in mind that Visual Studio uses its own local NPM and not the one you might have installed globally. This means that you need to run the above commands in “Developer Command Prompt for VS” and not in a standard command prompt.

Visual Studio Code

VS Code in its more recent versions supports proxy authentication and it can even help its extensions to use its proxy settings, but some extensions (e.g. NPM) still resist it (pun intended) and you might need to configure them independently. If for any reason you need to manualy set VS Code’s proxy please refer to: https://code.visualstudio.com/docs/setup/network

Git

Starting from version 1.7.10, Git supports NTLM proxy authentication. You can find the commit here. Although you still need to instruct Git to use your proxy, because it does not detect it from your OS. To do so, you can point to your proxy server’s URL and Git will do the rest. The following bash command is setting the proxy URL to Fiddler’s default URL, but you can use your company’s proxy server too. You can either point to Fiddler to take care of proxy authentication using the following command.

$ git config --global http.proxy http://localhost:8888

Or directly point to your corporate proxy and include your username. Git can remember your credentials using Windows Credential Manager, but you need to make sure that your Git client is configured to use the credential manager and you are using v. 2.8.0 or higher.

$ git config --global http.https://github.axa.com.proxy http://[proxyuser@]<proxyhost>:<port>

To make sure Git is using Windows Credential Manager run the following command.

$ git config --global credential.helper wincred
How to fix proxy authentication issues in your development environment

A shorter way to initialize Lists in C#

Threre is a trick that I wanted to show you today. A trick you can do initialize your lists a bit easier. Of course there’s some plumbing to be done.

Let’s assume you have a class definition like the following.

public class Foo
{
    public int A { get; set; }
    public int B { get; set; }
    public int C { get; set; }
    void Foo(int a, int b, int c)
    {
        A = a;
        B = b;
        C = c;
    }
}

You would typically initialize a list of Foo type like the following.

var foos = new List<Foo>();
foos.Add(new Foo(1, 2, 3));
foos.Add(new Foo(4, 5, 6));
// and more ...

But if you write and extension method like below.

public static class FooExtensions
{
    public static void Add(this IList<Foo>  list, int a, int b, int c) => list.Add(new Foo(a, b, c));
}

You can initialize your list like so:

var foos = new List<Foo>
{
    {1, 2, 3},
    {3, 4, 5},
    {6, 7, 8},
    // and more ...
};
A shorter way to initialize Lists in C#

How to use Ctrl+W to close a tab in Visual Studio

All the browsers I know give you the option to close tabs using Ctrl+W in addition to the standard Ctrl+F4 in Windows. I find Ctrl+W easier to reach than Ctrl+F4, but as a developer I switch back and forth between Visual Studio (or VS Code) and my browsers and sometimes by mistake I press Ctrl+W in Visual Studio. That’s why I looked around in Tools > Options to see if there is a way to change this shortcut in Visual Studio once and for all. It was right there waiting for me.

Here is the way to do it:

  1. Select “Tools > Options” from the menu.
  2. In the “Options” window look for “Keyboard” under “Environment“.
  3. In “Show commands containing:” textbox type “Window.CloseDocumentWindow“.
  4. There’s no surprise that the command is assigned “Ctrl+F4” by default. You need to remove it by clicking “Remove” button in front of “Shortcuts for selected command”. You might need to press a few times if there are more shortcuts assigned to it.
  5. Make sure “Global” is selected for “Use new shortcut in:“.
  6. Press “Ctrl+W” in “Press shortcut keys:” box.
  7. Don’t forget to click “OK” at the end to save your changes.

At the end, you should have something like the following picture.

Visual Studio Options - Keyboard shortcut for Windows.CloseDocumentWindow is set to Ctrl_W (Global)
Visual Studio – Options – Keyboard
How to use Ctrl+W to close a tab in Visual Studio

How to use PowerShell Invoke-WebRequest behind corporate proxy

Corportate proxies are one of the productivity killers for developers. They are not well supported in every utility and framework and each tool has its own litrature to set proxy settings. To add salt to the injury, not every tool supports NTLM authentication well which is quite common in many proxies. Companies have to sometimes make exception rules in proxy settings that can further complexify matters.

In case of PowerShell you do not have to worry much. Let’s see how you can set proxy for Invoke-WebRequest for example. Other commands usually support proxy settings similarly.

$pass = ConvertTo-SecureString "P@ssw0rd" -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential -ArgumentList "contoso\george", $pass
Invoke-WebRequest https://registry.npmjs.org/express -Proxy "http://proxy.contoso.com:8080" -ProxyCredential $cred

In line 1, we store the password in a SecureString object. In Line 2, we create a new PSCredentual object by providing the username and password and finally in Line 3, we call Invoke-WebRequest using -Proxy and -ProxyCrendential parameters.

Let me give you another alternative. Did you know you can also ask the proxy settings required for a URL from your OS and even use the current user’s credentials?

$dest = "https://registry.npmjs.org/express"
$proxy = ([System.Net.WebRequest]::GetSystemWebproxy()).GetProxy($dest)
Invoke-WebRequest https://registry.npmjs.org/express -Proxy $proxy -ProxyUseDefaultCredentials
How to use PowerShell Invoke-WebRequest behind corporate proxy

Custom Component vs Script Component

Why this comparison?

Recently while working on a (relatively) huge cloud migration project, I had the opportunity to brush up on my SSIS knowledge that was getting ancient. I had to work on several SSIS projects and pave the way for my fellow developers so we could all move faster.

When working on SSIS projects, often times there is need to write logic that cannot be achieved by using out-of-the-box components from the toolbox. In those cases the first option that automatically comes to mind is Script Component. This component lets you write code in your language of choice to act as a source, destination or transformation. You always also have the option of making your own shiny custom component. It is generally believed that making a custom component is usually more complicated and it takes more time so most of the times developers use Script Component throughout the project and it leads to duplicating code that is not maintainable and every change is hard to track to a point that in larger projects no one is really sure how each part works anymore 🙂

What’s wrong with Script Component

Script Component looks pretty good in theory. It gives you the possibility to embed code in a data flow. It can act as a source, destination or a transformation. The problem is in the way the code is embedded in the package. If you open the XML content of a package that has a Script Component is it, you will notice that it is actually a full Visual Studio project with a random generated namespace and some generated code to make it easy to interact with fields through typed properties. When you double click on a Script Component, SSDT decodes that projects and (presumably) writes all the files of that project in a temporary folder and opens it in a new instance of Visual Studio. When you close that project after applying your changes, it has do do the reverse. If you put a break-point somewhere in that project, then again it has to do all that again. So, to sum it up:

  • Script Components take large space in SSIS packages and if you create several Script Components in a package, it can dramatically increase the size of the package hence slowing down development and stability of Visual Studio and hinder productivity
  • Each Script Component is an independent project that makes debugging them harder specially that they are encoded and embedded within the package’s XML.
  • The fact that all Script Components are encoded and embedded within the XML file will make it really hard to do SSIS development in a team and it is very difficult to trace a change no matter which source control software you use. Imagine comparing thousands of lines of encoded XML to see where some logic might have been changed in some lines of scripts.
  • The debugger in Visual Studio might not work properly (even crash) when you want to debug multiple Script Components. It is also not possible to have breakpoints in more than one Script Component
  • Readability is another big issue. You will not be able to easily skim through your Script Components to understand the logic. Imagine you will need to open each one by one slowly and if you miss something, you will need to back again to a previous one. It is not like a typical Visual Studio project where you can easily navigate through code with several out-of-the-box features.
  • The name of the columns you get in the Script Component might not be exactly the same as original columns, if you decide to use generated typed properties.
  • It is not possible to reuse the logic that is inside a script component, unless if you externalize it in another assembly and in that case managing versions of that external assembly can become tedious, because you might need to open every single Script Component that references that assembly should the version change and if you by mistake do a breaking change you will not find out during compile time. You will need to test every single script component using that assembly and there is no easy way to find out which Script is referencing that. so, in short you will need a full end to end test to find an issue that you could be found in a compile in other types of projects.
  • It is not easy to duplicate a Script Component. I know that it is against DRY rule to duplicate code, but lets say due to above limitation in special scenario you decide to do so, in that case because the assembly name in those Script Components will be the same, during compile time one of will replace the other without SSDT giving you an error or even a warning.
  • This one is actually a bug. If you use more recent versions of C# language (I don’t know for VB, but probably the same) none of your break point will ever be hit. So, it is better to forget about those shiny features that came with C# 6.0+ like simplified dictionary initializer (i.e. { ["key"] = value }).

What can justify using a Script Component?

In my opinion with all the above problems around Script Components, the only reason you should use it is when you need to create a one off component as part of your data flow or when your are going to have just a few minimal Script Components in your package. In all other cases that I have experienced it is better to create a custom component. You might be thinking that building a custom component can be difficult or you might have already tried to find your way around MSDN to figure out how to do it, but got confused with the minimal explanation and over simplified examples that exists. But, trust me on this, it is not that difficult if you do it right. In my next post, I’m going to show you just that, but with a clear step by step guide and a real world example.

Custom Component vs Script Component