How to fix proxy authentication issues in your development environment

I have written about proxy authentication before and although this annoying problem has been around, many tools still don’t support it well. Specially when it comes to NTLM flavor of it that has been developed by Microsoft long time ago and even many Microsoft tools did not support it until recently.

Proxy servers in my opinion are not the best tools to control internet access. Security benefit they bring us are not much compared to productivity they take away from developers.

In the past I have used many different techniques to overcome this issue (e.g. CNTLM, custom local proxy and more) but the best way that almost always works reliably and securely is to run Fiddler with its “Capture Traffic” turned off (unless you need it for other reasons) and point from your blocked app to it. This way:

  1. Fiddler won’t be capturing all the traffic and it will only proxy the traffic for the applications that are specifically pointing to it.
  2. It takes care of authentication on behalf of you without worrying to store your credentials somewhere that is not safe or having to update your credentials when they change.
  3. You can always close Fiddler when you want to cut the traffic

In the rest of this blog post I will show how you can configure every development tool that I know (and remember) to point to Fiddler as a proxy.

Keep in mind that it doesn’t have to be Fiddler, any other local proxy that can take care of authentication on behalf of the logged in user can do the same.

Please let me know if I have forgotten a developer tool or if you know how to configure another tool and what others to know it. I will add it to the list as soon as possible.

Nuget Package Manager

At this moment Nuget Package Manager does not understand proxy authentication and you need to change its configuration to use your local proxy server (Fiddler) to take care of authentication. The configuration file is located in

%USERPROFILE%\AppData\Roaming\NuGet\NuGet.Config

<configuration>
  <config>
    <add key="http_proxy" value="http://localhost:8888" />
  </config>
</configuration>

Node Package Manager (NPM)

At this moment Nuget Package Manager does not understand proxy authentication and you need to change its configuration to use your local proxy server (Fiddler) to take care of authentication.Run the following command to know if your NPM is using any proxy.

Run the following command to know your current proxy setting (if any).

npm config get proxy

To set Fiddler as your proxy setting use the following command.

npm config set proxy http://localhost:8888

You have the possibility to point to your corporate proxy server using the following syntax.

npm config set proxy http://{domain\username-url-encoded}:{password-url-encoded}@{proxy-domain-name}:{port}

But keep in mind that if you directly point to your corporate proxy, every time your password changes you need to run the above command again to update the setting.

Side: To URL encode your username or password you can use encodeURIComponent in your browser’s Console. For example:

>> encodeURIComponent('domain\\username')
"domain%5Cusername"
>> encodeURIComponent("P@ssw0rd!")
"P%40ssw0rd!"

Visual Studio’s Task Runner (NPM in Visual Studio)

Whatever I explained about configuring NPM’s proxy setting previously is still applicable here, but you need to keep in mind that Visual Studio uses its own local NPM and not the one you might have installed globally. This means that you need to run the above commands in “Developer Command Prompt for VS” and not in a standard command prompt.

Visual Studio Code

VS Code in its more recent versions supports proxy authentication and it can even help its extensions to use its proxy settings, but some extensions (e.g. NPM) still resist it (pun intended) and you might need to configure them independently. If for any reason you need to manualy set VS Code’s proxy please refer to: https://code.visualstudio.com/docs/setup/network

Git

Starting from version 1.7.10, Git supports NTLM proxy authentication. You can find the commit here. Although you still need to instruct Git to use your proxy, because it does not detect it from your OS. To do so, you can point to your proxy server’s URL and Git will do the rest. The following bash command is setting the proxy URL to Fiddler’s default URL, but you can use your company’s proxy server too. You can either point to Fiddler to take care of proxy authentication using the following command.

$ git config --global http.proxy http://localhost:8888

Or directly point to your corporate proxy and include your username. Git can remember your credentials using Windows Credential Manager, but you need to make sure that your Git client is configured to use the credential manager and you are using v. 2.8.0 or higher.

$ git config --global http.https://github.axa.com.proxy http://[proxyuser@]<proxyhost>:<port>

To make sure Git is using Windows Credential Manager run the following command.

$ git config --global credential.helper wincred
How to fix proxy authentication issues in your development environment

A shorter way to initialize Lists in C#

Threre is a trick that I wanted to show you today. A trick you can do initialize your lists a bit easier. Of course there’s some plumbing to be done.

Let’s assume you have a class definition like the following.

public class Foo
{
    public int A { get; set; }
    public int B { get; set; }
    public int C { get; set; }
    void Foo(int a, int b, int c)
    {
        A = a;
        B = b;
        C = c;
    }
}

You would typically initialize a list of Foo type like the following.

var foos = new List<Foo>();
foos.Add(new Foo(1, 2, 3));
foos.Add(new Foo(4, 5, 6));
// and more ...

But if you write and extension method like below.

public static class FooExtensions
{
    public static void Add(this IList<Foo>  list, int a, int b, int c) => list.Add(new Foo(a, b, c));
}

You can initialize your list like so:

var foos = new List<Foo>
{
    {1, 2, 3},
    {3, 4, 5},
    {6, 7, 8},
    // and more ...
};
A shorter way to initialize Lists in C#

Development behind corporate proxy + authentication

Developing in companies that have proxy servers for developers can be frustrating in this age when every tool needs access to online resources and even parts of software development life cycle are cloud based. Proxy servers that require NTLM authentication just add to that frustration. NTLM is developed by Microsoft but many applications built by Microsoft do not support it or require some configuration and in worst cases some hacking to make it work. Below is a list of some the tools that developers might be using on a daily basis and what you need to do to make them connect via NTLM proxy. I keep adding more to the list as I encounter them.

  • Visual Studio Code (VSCode)
  • NPM
  • Visual Studio, Web Platform Installer and other .NET Applications

Visual Studio Code (VSCode)

VSCode 1.15 and up now supports NTLM proxy (finally Microsoft supported its own authentication protocol).

NPM

For NPM you have two options. Either to send proxy address and credentials in every single command you run or to set them in the global configuration of NPM. I recommend the former because it is more secure.

Set proxy in every command

When calling NPM command you can always use --proxy switch to set proxy for each command. The syntax for using this switch is the following.

--proxy username:password@proxyaddress:port

For example to use myproxy:8080 as proxy address and my-domain\reza as username and P@ssw0rd as password when calling the install command you can type the following.

npm install --proxy http://my-domain%5Creza:P%40ssw0rd@myproxy:8080

Please note that both username and password are URL encoded. You can use the following command in your browser’s developer tools to encode them.

encodeURI('my-domain\\reza')
encodeURI('P@ssw0rd')

Set proxy in NPM configuration

To set the proxy in the global configuration of NPM you need use the same format as above for sending the proxy address, username and password and use npm config set to store it in the configuration. For example to set the proxy address to myproxy:8080 and username to my-domain\reza and password to P@ssw0rd you can use the following command.

npm config set proxy http://my-domain%5Creza:P%40ssword@myproxy:8080

Visual Studio, Web Platform Installer and other .NET applications

To set the proxy for pretty much any .NET application, you need to put the following in the configuration file of that application.

<system.net>
    <defaultProxy useDefaultCredentials="true" enabled="true">
        <proxy bypassonlocal="true" proxyaddress="http://myproxy:8080" />
    </defaultProxy>
</system.net>

For Visual Studio I suggest also enabling IPV6 if the above configuration did not work as suggested by some other developers.

<system.net>
    <settings>
        <ipv6 enabled="true"/>
    </settings>
    <defaultProxy useDefaultCredentials="true" enabled="true">
        <proxy bypassonlocal="true" proxyaddress="http://myproxy:8080" />
    </defaultProxy>
</system.net>

For executable files the configuration file is named the same as executable’s file name but with .exe.config extension. For Visual Studio it is called devenv.exe.config and for Web Platform Installer it is WebPlatformInstaller.exe.config.

Postman

At the moment Postman (v7.36.1) does not support NTLM authentication and the only best way that you can make it work is by using Fiddler. Here is what you need to do, step by step:

  1. Install Fiddler and run it.
  2. Make sure Rules > Automatically Authenticate is selected. This will enable Fiddler to authenticate on behalf of Postman with your current user account.
  3. * In Postman, go to File > Settings and then Proxy and turn on Global Proxy Configuration
  4. For Proxy Type select both HTTP and HTTPS
  5. For the Proxy Server, use 127.0.0.1:8888 (If you you have changed the default port that Fiddler is listening on you will need to change it here as well).
  6. ** In Fiddler, go to Tools > Options... and then in HTTP tab, select Capture HTTPS CONNECTs and Decrypt HTTPS traffic and install the certificate when prompted.
  7. In Postman go to File > Settings > General, turn off SSLS certificate verification. You need to do this because currently, Postman does not support intermediate proxies.

* In step 6, instead of using Global Proxy Configuration you may also use Use System Proxy, but in that case you need to make sure in Fiddler, Capture Traffic is selected under File menu. This way Fiddler will capture all the HTTP traffic by setting Windows Proxy settings.

** You won’t need to do step 6 and 7 if you won’t be working with HTTPS URLs.

Development behind corporate proxy + authentication

How to warm-up SharePoint or other web applications and WCF (SOAP) services with PowerShell

There are many reasons you might want to warm-up a web application occasionally. It can be after a fresh deployment or on a regular basis after recycling application pools. Some times you might also need to warm-up SOAP services without going through front-end.

It might seems to be any easy task specially if you have PowerShell 3.0 or higher on your servers, but after Googling a while and reviewing some of the top hits I discovered that each solution is missing a part. Some only work in a single server scenario and some has forgotten that each HTTP response might contain links to scripts and images that we need to download and finally I could not find anything for SOAP services that just works. Long story short I decided to put together a simple script that just works and is easy to change to everyone’s needs.

Please note that my only assumption is you have PowerShell 3.0+ in your servers.

Currently the script takes care of the following tasks, but I will most likely improve it to cover other useful scenarios.

  • Calling SOAP operations and sending parameters and custom headers
  • Calling front-end URIs and downloading scripts and images that are local to the front-end
  • Logging to a configurable folder and file name
  • Cleaning up old log files

Currently, I have the following points in mind to improve the script.

  • Put configuration in a different file.
  • Improve function definitions (named and typed parameters and description).
  • Default values for parameters when it makes sense (e.g. log folder can be the current folder).
  • Support REST services.
#Parameters
$logFolder = "C:\Temp\logs"
$logFile = "$logFolder\Warmup_$(Get-Date Format "MM-dd-yyyy_hh-mm-ss").log"
$timeoutSec = 10
$headers = @{
SOAPAction= "";
"Accept-Encoding" = "gzip,deflate";
"User-Agent" = "PowerShell/Warmup-WebApp"
}
$webRequests = @(
@{
SOAPAction = "http://tempuri.org/IService1/Operation1";
Body = @"
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/&quot; xmlns:tem="http://tempuri.org/"&gt;
<soapenv:Body>
<tem:Operation1>
<tem:Parameter1>1</tem:Parameter1>
<tem:Parameter2>2</tem:Parameter2>
</tem:Operation1>
</soapenv:Body>
</soapenv:Envelope>
"@
},
@{
SOAPAction = "http://tempuri.org/IService1/Operation2";
Body = @"
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/&quot; xmlns:tem="http://tempuri.org/"&gt;
<soapenv:Body>
<tem:Operation1>
<tem:Parameter1>1</tem:Parameter1>
<tem:Parameter2>2</tem:Parameter2>
</tem:Operation1>
</soapenv:Body>
</soapenv:Envelope>
"@
}
)
# Functions
function Write-Log([string]$level, [string]$text) {
$logTime = Get-Date Format "yyyy-MM-dd HH:mm:ss.fff"
Write-Host "[$level] $text"
if ($logFile -ne $null) {
"$logTime [$level] $text" | Out-File FilePath $Logfile Append
}
}
function Cleanup-LogFiles() {
$expiryDate = (Get-Date).AddMinutes(-7)
[bool]$logFileDeleted = $false
$filesToDelete = Get-Childitem $logFolder Include "*.log" Recurse | ? {$_.LastWriteTime -le $expiryDate} | % {
Remove-Item $_.FullName | out-null
Write-Log "Information" "File deleted successfully – $_"
$logFileDeleted = $true
}
if (-not $logFileDeleted) {
Write-Log "Information" "No old log file was found in $logFolder"
}
}
function Warmup-WcfService($serviceUrl) {
foreach($webRequest in $webRequests) {
$headers["SOAPAction"] = $webRequest.SOAPAction
Invoke-WcfService $serviceUrl $headers $webRequest.Body
}
}
function Invoke-WcfService($uri, $headers, $body) {
$invocationStartTime = Get-Date
try {
$result = Invoke-WebRequest Uri $uri Method Post UseDefaultCredentials ContentType "text/xml; charset=utf-8" Headers $headers body $body TimeoutSec $timeoutSec
$invocationDuration = (Get-Date).Subtract($invocationStartTime)
if ($result.StatusCode -eq 200) {
if ($result.Headers.ContainsKey("Content-Length")) {
$contentLength = $result.Headers["Content-Length"]
} else {
$contentLength = "$($result.RawContentLength) (raw)"
}
Write-Log "Information" "WCF operation invoked successfully – Operation: $($webRequest.SOAPAction)($variation), Duration: $($invocationDuration), Length: $contentLength"
} else {
Write-Log "Warning" "WCF operation failed – Operation: $($webRequest.SOAPAction)($variation), Status: $($result.StatusCode) $($result.StatusDescription), Length: $contentLength"
}
} catch {
Write-Log "Critical" "WCF operation failed – Operation: $($webRequest.SOAPAction)($variation), Exception: $_"
}
}
function Invoke-WebPage([string]$uri, [bool]$getImages, [bool]$getScripts) {
try {
$response = Invoke-WebRequest Uri $uri UseDefaultCredentials TimeoutSec $timeoutSec Proxy "http://localhost" ProxyUseDefaultCredentials
if ($response.StatusCode -eq 200) {
Write-Log "Information" "Web page invoked successfully – URL: $uri, Images: $($response.Images.Count), Scripts: $($response.Scripts.Count), Links: $($response.Links.Count)"
} else {
Write-Log "Warning" "Web page invocation failed – URL: $uri, Status: $($response.StatusCode) $($response.StatusDescription)"
}
} catch {
Write-Log "Critical" "Web page invocation failed – URL: $uri, Exception: $_"
}
if ($getImages) {
FetchResources $uri ($response.Images | ? { $_.src } | % { $_.src })
}
if ($getScripts) {
FetchResources $uri ($response.Scripts | ? { $_.src } | % { $_.src })
}
}
Function Fetch-Resources($baseUrl, $resources) {
[uri]$uri = $baseUrl
$rootUrl = $uri.Scheme + "://" + $uri.Authority
$counter = 0
foreach ($resUrl in $resources) {
if ($resUrl.StartsWith("http", [System.StringComparison]::OrdinalIgnoreCase)) {
$fetchUrl = $resUrl
} elseif (!$resUrl.StartsWith("/")) {
$fetchUrl = $rootUrl + "/" + $resUrl
} else {
$fetchUrl = $rootUrl + $resUrl
}
# Progress
Write-Progress Activity "Opening " Status $fetchUrl PercentComplete (($counter/$resources.Length)*100)
$counter++
# Execute
try {
$resp = Invoke-WebRequest UseDefaultCredentials UseBasicParsing Uri $fetchUrl TimeoutSec 120
Write-Log "Information" "Resource has been downloaded successfully – URL: $fetchUrl"
} catch {
Write-Log "Warning" "Resource failed to download – URL: $fetchUrl, Exception: $_"
}
}
Write-Progress Activity "Completed" Completed
}
# Execution
Write-Log "Information" "My Web Application Warmup Script – $(Get-Date)"
Write-Log "Information" "Cleaning up old log files…"
CleanupLogFiles
Write-Log "Information" "Warming up WCF Services…"
WarmupWcfServices
Write-Log "Information" "Warming up Front-End…"
Invoke-WebPage "http://domain.com/area1/Page1.aspx" $true $true
Invoke-WebPage "http://domain.com/area1/Page2.aspx" $true $true
Write-Log "Information" "My Web Application Warmup Script Completed."

view raw
Warmup-WebApp.ps1
hosted with ❤ by GitHub

I’m open to any suggestion and feature request. Please let me know if you found it useful or if have got something wrong.

How to warm-up SharePoint or other web applications and WCF (SOAP) services with PowerShell

How to read optional attributes in XML with default values

When we are reading data from XML data sources, it is very common that when a value is not given for an attribute or an element, we have to use a default value instead. Having to check for the existence of the node each and every time we are reading a value can be annoying and clutter the code. To address this issue I have made a few useful extension method that you can use copy and use in your code. These are written on top of XAttribute class, but you can write similar methods for other types and APIs pretty much the same way.

public static class XmlExtentions
{
    public static string ValueOrDefault(this XAttribute attr)
    {
        return ValueOrDefault(attr, string.Empty);
    }
    public static string ValueOrDefault(this XAttribute attr, string defaultValue)
    {
        return attr == null ? defaultValue : attr.Value;
    }

    public static T ValueOrDefault<T>(this XAttribute attr, T defaultValue) where T : struct
    {
        if (attr == null) { return defaultValue; }
        var value = attr.Value;
        var convertedValue = typeof(T).IsEnum ? Enum.Parse(typeof(T), value) : Convert.ChangeType(value, typeof(T));
        if (convertedValue == null) { return defaultValue; }
        return (T) convertedValue;
    }

    public static T ValueOrDefault<T>(this XAttribute attr, Func<T> defaultDelegate) where T : struct
    {
        if (attr == null) { return defaultDelegate(); }
        var convertedValue = Convert.ChangeType(attr.Value, typeof(T));
        if (convertedValue == null) { return defaultDelegate(); }
        return (T)convertedValue;
    }
}

Once you get it in place. It is pretty easy to use. Assume that we have the following XML document:

<Configuration>
    <Search Enabled="True" RankingAlghorithm="BS77" LowMark="1000" HighMark="2000">
        <Source Name="Orders" Type="WebService" />
        <Source Name="Products" Type="REST" />
    </Search>
<Configuration>

As is shown below, you can read any type of values exactly the same way.

// reading a bool
var isEnabled = sourceNode.Attribute("Enabled").ValueOrDefault(false);

// reading a string
var scopeName = sourceNode.Attribute("Name").ValueOrDefault("Default");

// reading a string and using empty string (string.Empty) as default value
var scopeName = sourceNode.Attribute("Name").ValueOrDefault();

// reading an enum
var sourceType = sourceNode.Attribute("Type").ValueOrDefault(SearchSources.Html);

// reading an integer
var lowMark = sourceNode.Attribute("LowMark").ValueOrDefault(100);

// Using a delegate to provide the default value
var highMark = sourceNode.Attribute("HighMark").ValueOrDefault(()=> App.GetUserSetting("Search.HighMark"));
How to read optional attributes in XML with default values