Development behind corporate proxy + authentication

Developing in companies that have proxy servers for developers can be frustrating in this age when every tool needs access to online resources and even parts of software development life cycle are cloud based. Proxy servers that require NTLM authentication just add to that frustration. NTLM is developed by Microsoft but many applications built by Microsoft do not support it or require some configuration and in worst cases some hacking to make it work. Below is a list of some the tools that developers might be using on a daily basis and what you need to do to make them connect via NTLM proxy. I keep adding more to the list as I encounter them.

  • Visual Studio Code (VSCode)
  • NPM
  • Visual Studio, Web Platform Installer and other .NET Applications

Visual Studio Code (VSCode)

VSCode 1.15 and up now supports NTLM proxy (finally Microsoft supported its own authentication protocol).

NPM

For NPM you have two options. Either to send proxy address and credentials in every single command you run or to set them in the global configuration of NPM. I recommend the former because it is more secure.

Set proxy in every command

When calling NPM command you can always use --proxy switch to set proxy for each command. The syntax for using this switch is the following.

--proxy username:password@proxyaddress:port

For example to use myproxy:8080 as proxy address and my-domain\reza as username and P@ssw0rd as password when calling the install command you can type the following.

npm install --proxy http://my-domain%5Creza:P%40ssw0rd@myproxy:8080

Please note that both username and password are URL encoded. You can use the following command in your browser’s developer tools to encode them.

encodeURI('my-domain\\reza')
encodeURI('P@ssw0rd')

Set proxy in NPM configuration

To set the proxy in the global configuration of NPM you need use the same format as above for sending the proxy address, username and password and use npm config set to store it in the configuration. For example to set the proxy address to myproxy:8080 and username to my-domain\reza and password to P@ssw0rd you can use the following command.

npm config set proxy http://my-domain%5Creza:P%40ssword@myproxy:8080

Visual Studio, Web Platform Installer and other .NET applications

To set the proxy for pretty much any .NET application, you need to put the following in the configuration file of that application.

<system.net>
    <defaultProxy useDefaultCredentials="true" enabled="true">
        <proxy bypassonlocal="true" proxyaddress="http://myproxy:8080" />
    </defaultProxy>
</system.net>

For Visual Studio I suggest also enabling IPV6 if the above configuration did not work as suggested by some other developers.

<system.net>
    <settings>
        <ipv6 enabled="true"/>
    </settings>
    <defaultProxy useDefaultCredentials="true" enabled="true">
        <proxy bypassonlocal="true" proxyaddress="http://myproxy:8080" />
    </defaultProxy>
</system.net>

For executable files the configuration file is named the same as executable’s file name but with .exe.config extension. For Visual Studio it is called devenv.exe.config and for Web Platform Installer it is WebPlatformInstaller.exe.config.

Postman

At the moment Postman (v7.36.1) does not support NTLM authentication and the only best way that you can make it work is by using Fiddler. Here is what you need to do, step by step:

  1. Install Fiddler and run it.
  2. Make sure Rules > Automatically Authenticate is selected. This will enable Fiddler to authenticate on behalf of Postman with your current user account.
  3. * In Postman, go to File > Settings and then Proxy and turn on Global Proxy Configuration
  4. For Proxy Type select both HTTP and HTTPS
  5. For the Proxy Server, use 127.0.0.1:8888 (If you you have changed the default port that Fiddler is listening on you will need to change it here as well).
  6. ** In Fiddler, go to Tools > Options... and then in HTTP tab, select Capture HTTPS CONNECTs and Decrypt HTTPS traffic and install the certificate when prompted.
  7. In Postman go to File > Settings > General, turn off SSLS certificate verification. You need to do this because currently, Postman does not support intermediate proxies.

* In step 6, instead of using Global Proxy Configuration you may also use Use System Proxy, but in that case you need to make sure in Fiddler, Capture Traffic is selected under File menu. This way Fiddler will capture all the HTTP traffic by setting Windows Proxy settings.

** You won’t need to do step 6 and 7 if you won’t be working with HTTPS URLs.

Developing NodeJS apps using AngularJS 4 in macOS

Well, that is one long title for a blog post! but, bare with me I’m going straight to the point. Recently I’ve been wanting to dip my toes in the AngularJS 4 world and to make it more exciting I thought maybe I will do it on a Mac using NodeJS and VSCode. Now that VSCode is getting more and more popular sharing my experience here might actually help someone and a future me to have an easier ride than mine today.

The ingredients

  • A computer with macOS on it. Any version would do.
  • Homebrew
  • VSCode
  • Git client for Mac
  • NodeJS
  • Half an hour of time!

Preparation

We first need to get the environment ready before we start to develop something and believe it or not there are several ways you can do that. After studying and trying out different ways, this is the best way I found that is the safest, fastest, easiest, most repeatable and undoable, and yet inline with best practices of Apple in Mac world!

  1. Install Homebrew if you don’t already have it.
  2. Install NPM
  3. Install VSCode or any other code editor / IDE
  4. Install Git client
  5. Install NodeJS
  6. Make a folder
  7. Start VSCode

1. Install Homebrew

Although NodeJS website provides an installer for NPM,  I suggest using Homebrew to install NPM and any other open source package you might need in you Mac. NPM installer needs admin rights and it will put files in folders that only admin users have access to. You don’t want any 3rd party application tinkering with those areas do you?! plus it will be easier to cleanup afterwards should you need to remove NPM for some reason in future.

I recommend you visit Homebrew’s official website https://brew.sh to know more about it. At the time of writing this blog post, you just need to open Terminal and run the following command to execute Homebrew’s installation script. The script will explain what it does and then pauses for your confirmation.


ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

2. Install NPM

Now that you have installed Homebrew, installing NPM is as simple as typing the following command in the Terminal window. It will install the latest stable version.


brew install node

After the installation is done, you can type the following command to see which version is installed.


node -v

If you need to update Node and NPM in future you can use brew upgrade node and to uninstall them you can use brew uninstall node.

3. Install Visual Studio Code

This step is pretty easy. You just need to download and install Visual Studio Code, Microsoft’s lightweight code editor from its official website and install it like any other application. Here is the official download page: https://code.visualstudio.com/download

Directions

There are two ways you can start to write a new application by AngularJS. You can either manually set up the project by creating the required boilerplate files and writing configurations which can really help you understand what is going on behind the scene or you can use the Angular CLI to do it for you and this way you save some time. I’m going to explain both methods here and first I am going to start with the manual method. Instead of explaining the manual method, I’m going to show you how to use Angular CLI because that’s the way you’re most probably going to do it in future. Then I will explain all the files and configurations in such a way that you can even do it manually if you wish to.

[To be continued!]

 

 

How to mimic Pivot Table or Categories in Number 3+

Perhaps you are already aware that Apple has cut many features from latest versions of Numbers and their other MS Office like products. One of the most popular features that has been removed from Numbers is called Categories. By activating Categories on a table you could aggregate data and summarize values see an overview of what matters to you. Recently I was working on a list of values and since the list was not bigger than what Numbers can handle and I didn’t need the speed and power of Microsoft Excel. I though I will give Numbers a try. I still like the fact that you have a free canvas in Numbers that allows you to put put many tables independent from each other in one page. I think it is the only advantage of Numbers to its alternatives. Otherwise with just a few dollars per months you could have access to the latest version of Microsoft Office regardless of your OS or even directly in your browser plus Microsoft gives you a terabyte of online storage! OK, let’s get back to work before I change your mind 😉

Let’s start with an example like the following.

A B
1 Category Amount
2 Blue 11.20
3 Red 15.89
4 Red 10.30
5 Orange 32.12
6 Green 15.39
7 Blue 10.18
8 Green 24.76
9 Green 89.31
10 Orange 8.75
11 Blue 15.28

The first column is just containing the order of each row, the second indicates the category of each row. This could be anything like category of expenses in an expense report. It is actually the column that we are going to aggregate. The third column contains the values. It could be the amount of each expense for example.

Now, let’s say you want to aggregate it to the following table. But you want Numbers to do it for you so each time a series of row is added to the table you don’t need to calculate everything manually.

Category Sum
Blue 36.66
Red 26.19
Orange 40.87
Green 129.46

Since I want to do the aggregation in the same table, I need to find a way to detect distinct categories. Then per each category I need to calculate and display the sum of each category in front of it. If I display only the first occurrence of each category and hide other occurrences the problem would be solved. Let’s split the problem to smaller pieces as usual. Now the first question would be how do we find distinct categories?

Step 1 – Detecting distinct categories

Let’s add a new column called “Test” then put the following formula in the first cell of the column below the header: =COUNTIF(A$1:A1,A2) in other words you need to put it in C2 cell. Now click at the bottom of the cell’s rectangle and drag it to the bottom of the table. This will fill remaining cells of the column. The result will be the following table.

A B C
1 Category Amount Test
2 Blue 11.20 0
3 Red 15.89 0
4 Red 10.30 1
5 Orange 32.12 0
6 Green 15.39 0
7 Blue 10.18 1
8 Green 24.76 1
9 Green 89.31 2
10 Orange 8.75 1
11 Blue 15.28 2

As you can see, you only get zeros for the first occurrence of each category and that means we can detect them with an IF. Now, let’s put the following formula in cell C2 instead: =IF(COUNTIF(A$1:A1,A2)=0,"*","") and as before drag the corner of the cell until the bottom of the table which will give us the following result.

A B C
1 Category Amount Test
2 Blue 11.20 *
3 Red 15.89 *
4 Red 10.30
5 Orange 32.12 *
6 Green 15.39 *
7 Blue 10.18
8 Green 24.76
9 Green 89.31
10 Orange 8.75
11 Blue 15.28

Now for first occurrence of each category we are displaying “*” in column C.It means that we are able to detect the first occurrence of each category and display an arbitrary text in it. What about displaying SUM of each category there. This would be the final solution.

My final solution

In this step we are going to replace the “*” in the formula with SUMIF(Category,C2,Amount) this way the SUM of each category will be calculate in front of each occurrence of that category. All you have to do is to put the following formula in C2, then drag the bottom-right corner of the cell until the bottom of the column.
=IF(COUNTIF(C$1:C1,C2)=0,SUMIF(Category,A2,Amount),””)
The resulting table will be similar to the following. I just renamed the Test column to “Total per category”. You can use formatting to distinguish the aggregate row or even hide other rows if you like.

A B C
1 Category Amount Total per category
2 Blue 11.20 36.66
3 Red 15.89 26.19
4 Red 10.30
5 Orange 32.12 40.87
6 Green 15.39 129.46
7 Blue 10.18
8 Green 24.76
9 Green 89.31
10 Orange 8.75
11 Blue 15.28

How to warm-up SharePoint or other web applications and WCF (SOAP) services with PowerShell

There are many reasons you might want to warm-up a web application occasionally. It can be after a fresh deployment or on a regular basis after recycling application pools. Some times you might also need to warm-up SOAP services without going through front-end.

It might seems to be any easy task specially if you have PowerShell 3.0 or higher on your servers, but after Googling a while and reviewing some of the top hits I discovered that each solution is missing a part. Some only work in a single server scenario and some has forgotten that each HTTP response might contain links to scripts and images that we need to download and finally I could not find anything for SOAP services that just works. Long story short I decided to put together a simple script that just works and is easy to change to everyone’s needs.

Please note that my only assumption is you have PowerShell 3.0+ in your servers.

Currently the script takes care of the following tasks, but I will most likely improve it to cover other useful scenarios.

  • Calling SOAP operations and sending parameters and custom headers
  • Calling front-end URIs and downloading scripts and images that are local to the front-end
  • Logging to a configurable folder and file name
  • Cleaning up old log files

Currently, I have the following points in mind to improve the script.

  • Put configuration in a different file.
  • Improve function definitions (named and typed parameters and description).
  • Default values for parameters when it makes sense (e.g. log folder can be the current folder).
  • Support REST services.
#Parameters
$logFolder = "C:\Temp\logs"
$logFile = "$logFolder\Warmup_$(Get-Date –Format "MM-dd-yyyy_hh-mm-ss").log"
$timeoutSec = 10
$headers = @{
SOAPAction= "";
"Accept-Encoding" = "gzip,deflate";
"User-Agent" = "PowerShell/Warmup-WebApp"
}
$webRequests = @(
@{
SOAPAction = "http://tempuri.org/IService1/Operation1";
Body = @"
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/&quot; xmlns:tem="http://tempuri.org/"&gt;
<soapenv:Body>
<tem:Operation1>
<tem:Parameter1>1</tem:Parameter1>
<tem:Parameter2>2</tem:Parameter2>
</tem:Operation1>
</soapenv:Body>
</soapenv:Envelope>
"@
},
@{
SOAPAction = "http://tempuri.org/IService1/Operation2";
Body = @"
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/&quot; xmlns:tem="http://tempuri.org/"&gt;
<soapenv:Body>
<tem:Operation1>
<tem:Parameter1>1</tem:Parameter1>
<tem:Parameter2>2</tem:Parameter2>
</tem:Operation1>
</soapenv:Body>
</soapenv:Envelope>
"@
}
)
# Functions
function Write-Log([string]$level, [string]$text) {
$logTime = Get-Date –Format "yyyy-MM-dd HH:mm:ss.fff"
Write-Host "[$level] $text"
if ($logFile -ne $null) {
"$logTime [$level] $text" | Out-File –FilePath $Logfile –Append
}
}
function Cleanup-LogFiles() {
$expiryDate = (Get-Date).AddMinutes(-7)
[bool]$logFileDeleted = $false
$filesToDelete = Get-Childitem $logFolder –Include "*.log" –Recurse | ? {$_.LastWriteTime -le $expiryDate} | % {
Remove-Item $_.FullName | out-null
Write-Log "Information" "File deleted successfully – $_"
$logFileDeleted = $true
}
if (-not $logFileDeleted) {
Write-Log "Information" "No old log file was found in $logFolder"
}
}
function Warmup-WcfService($serviceUrl) {
foreach($webRequest in $webRequests) {
$headers["SOAPAction"] = $webRequest.SOAPAction
Invoke-WcfService $serviceUrl $headers $webRequest.Body
}
}
function Invoke-WcfService($uri, $headers, $body) {
$invocationStartTime = Get-Date
try {
$result = Invoke-WebRequest –Uri $uri –Method Post –UseDefaultCredentials –ContentType "text/xml; charset=utf-8" –Headers $headers –body $body –TimeoutSec $timeoutSec
$invocationDuration = (Get-Date).Subtract($invocationStartTime)
if ($result.StatusCode -eq 200) {
if ($result.Headers.ContainsKey("Content-Length")) {
$contentLength = $result.Headers["Content-Length"]
} else {
$contentLength = "$($result.RawContentLength) (raw)"
}
Write-Log "Information" "WCF operation invoked successfully – Operation: $($webRequest.SOAPAction)($variation), Duration: $($invocationDuration), Length: $contentLength"
} else {
Write-Log "Warning" "WCF operation failed – Operation: $($webRequest.SOAPAction)($variation), Status: $($result.StatusCode) $($result.StatusDescription), Length: $contentLength"
}
} catch {
Write-Log "Critical" "WCF operation failed – Operation: $($webRequest.SOAPAction)($variation), Exception: $_"
}
}
function Invoke-WebPage([string]$uri, [bool]$getImages, [bool]$getScripts) {
try {
$response = Invoke-WebRequest –Uri $uri –UseDefaultCredentials –TimeoutSec $timeoutSec –Proxy "http://localhost" –ProxyUseDefaultCredentials
if ($response.StatusCode -eq 200) {
Write-Log "Information" "Web page invoked successfully – URL: $uri, Images: $($response.Images.Count), Scripts: $($response.Scripts.Count), Links: $($response.Links.Count)"
} else {
Write-Log "Warning" "Web page invocation failed – URL: $uri, Status: $($response.StatusCode) $($response.StatusDescription)"
}
} catch {
Write-Log "Critical" "Web page invocation failed – URL: $uri, Exception: $_"
}
if ($getImages) {
Fetch–Resources $uri ($response.Images | ? { $_.src } | % { $_.src })
}
if ($getScripts) {
Fetch–Resources $uri ($response.Scripts | ? { $_.src } | % { $_.src })
}
}
Function Fetch-Resources($baseUrl, $resources) {
[uri]$uri = $baseUrl
$rootUrl = $uri.Scheme + "://" + $uri.Authority
$counter = 0
foreach ($resUrl in $resources) {
if ($resUrl.StartsWith("http", [System.StringComparison]::OrdinalIgnoreCase)) {
$fetchUrl = $resUrl
} elseif (!$resUrl.StartsWith("/")) {
$fetchUrl = $rootUrl + "/" + $resUrl
} else {
$fetchUrl = $rootUrl + $resUrl
}
# Progress
Write-Progress –Activity "Opening " –Status $fetchUrl –PercentComplete (($counter/$resources.Length)*100)
$counter++
# Execute
try {
$resp = Invoke-WebRequest –UseDefaultCredentials –UseBasicParsing –Uri $fetchUrl –TimeoutSec 120
Write-Log "Information" "Resource has been downloaded successfully – URL: $fetchUrl"
} catch {
Write-Log "Warning" "Resource failed to download – URL: $fetchUrl, Exception: $_"
}
}
Write-Progress –Activity "Completed" –Completed
}
# Execution
Write-Log "Information" "My Web Application Warmup Script – $(Get-Date)"
Write-Log "Information" "Cleaning up old log files…"
Cleanup–LogFiles
Write-Log "Information" "Warming up WCF Services…"
Warmup–WcfServices
Write-Log "Information" "Warming up Front-End…"
Invoke-WebPage "http://domain.com/area1/Page1.aspx" $true $true
Invoke-WebPage "http://domain.com/area1/Page2.aspx" $true $true
Write-Log "Information" "My Web Application Warmup Script Completed."

view raw
Warmup-WebApp.ps1
hosted with ❤ by GitHub

I’m open to any suggestion and feature request. Please let me know if you found it useful or if have got something wrong.

How to create search query content source in SharePoint using PowerShell

They are called Content Sources in the graphical user interface of SharePoint and Search Query Content Source when using PowerShell. A content source is basically pointing to a source of data that can be indexed in the search database. Because the data is indexed, when later users run queries the result is returned almost instantaneous. It is very easy to add a content source by using SharePoint Central Administration, but you might need to create a long list of content sources in a batch for example as part of a disaster recovery process or many other reasons. Here is one way I do it in a PowerShell script.

$contentSourceConfig = @(
    @{
        Name = "Articles"
        StartAddresses = "http://mydomain1/site1/fr,http://mydomain1/site1/nl,http://mydomain1/site1/en"
    },
    @{
        Name = "Technical"
        StartAddresses = "http://mydomain1/site2/fr,http://mydomain1/site2/nl,http://mydomain1/site2/en"
    }
)

foreach($config in $contentSourceConfig) {
    $contentSourceName = $config.Name
    $startAddresses = $config.StartAddresses
    $contentSource = Get-SPEnterpriseSearchCrawlContentSource $contentSourceName -SearchApplication $ssa -ErrorAction SilentlyContinue
    if ($contentSource) { $contentSource.Delete() }
    $contentSource = New-SPEnterpriseSearchCrawlContentSource $contentSourceName -Type "Web" -StartAddresses $startAddresses -SearchApplication $ssa
    Write-Host "Search crawl content source: $contentSourceName created successfully."
}

As you see in the above example I have put all the required configuration in one array of custom objects. If you need to add more content sources you may simply copy-past one and change the properties. This way the configuration is separated and easier to read or change while still it is in the same file with the script so you won’t have to rely on extra XML file(s) to put beside the script.

If you are more comfortable by separating the configuration from the script. You can cut the first part and past it to another script file or use the XML approach.

For more details about the cmdlets I’ve used I suggest you have a look on the following MSDN articles: