PowerShell

How to use PowerShell Invoke-WebRequest behind corporate proxy

Corportate proxies are one of the productivity killers for developers. They are not well supported in every utility and framework and each tool has its own litrature to set proxy settings. To add salt to the injury, not every tool supports NTLM authentication well which is quite common in many proxies. Companies have to sometimes make exception rules in proxy settings that can further complexify matters.

In case of PowerShell you do not have to worry much. Let’s see how you can set proxy for Invoke-WebRequest for example. Other commands usually support proxy settings similarly.

$pass = ConvertTo-SecureString "P@ssw0rd" -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential -ArgumentList "contoso\george", $pass
Invoke-WebRequest https://registry.npmjs.org/express -Proxy "http://proxy.contoso.com:8080" -ProxyCredential $cred

In line 1, we store the password in a SecureString object. In Line 2, we create a new PSCredentual object by providing the username and password and finally in Line 3, we call Invoke-WebRequest using -Proxy and -ProxyCrendential parameters.

Let me give you another alternative. Did you know you can also ask the proxy settings required for a URL from your OS and even use the current user’s credentials?

$dest = "https://registry.npmjs.org/express"
$proxy = ([System.Net.WebRequest]::GetSystemWebproxy()).GetProxy($dest)
Invoke-WebRequest https://registry.npmjs.org/express -Proxy $proxy -ProxyUseDefaultCredentials

How to warm-up SharePoint or other web applications and WCF (SOAP) services with PowerShell

There are many reasons you might want to warm-up a web application occasionally. It can be after a fresh deployment or on a regular basis after recycling application pools. Some times you might also need to warm-up SOAP services without going through front-end.

It might seems to be any easy task specially if you have PowerShell 3.0 or higher on your servers, but after Googling a while and reviewing some of the top hits I discovered that each solution is missing a part. Some only work in a single server scenario and some has forgotten that each HTTP response might contain links to scripts and images that we need to download and finally I could not find anything for SOAP services that just works. Long story short I decided to put together a simple script that just works and is easy to change to everyone’s needs.

Please note that my only assumption is you have PowerShell 3.0+ in your servers.

Currently the script takes care of the following tasks, but I will most likely improve it to cover other useful scenarios.

  • Calling SOAP operations and sending parameters and custom headers
  • Calling front-end URIs and downloading scripts and images that are local to the front-end
  • Logging to a configurable folder and file name
  • Cleaning up old log files

Currently, I have the following points in mind to improve the script.

  • Put configuration in a different file.
  • Improve function definitions (named and typed parameters and description).
  • Default values for parameters when it makes sense (e.g. log folder can be the current folder).
  • Support REST services.


#Parameters
$logFolder = "C:\Temp\logs"
$logFile = "$logFolder\Warmup_$(Get-Date –Format "MM-dd-yyyy_hh-mm-ss").log"
$timeoutSec = 10
$headers = @{
SOAPAction= "";
"Accept-Encoding" = "gzip,deflate";
"User-Agent" = "PowerShell/Warmup-WebApp"
}
$webRequests = @(
@{
SOAPAction = "http://tempuri.org/IService1/Operation1";
Body = @"
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/&quot; xmlns:tem="http://tempuri.org/"&gt;
<soapenv:Body>
<tem:Operation1>
<tem:Parameter1>1</tem:Parameter1>
<tem:Parameter2>2</tem:Parameter2>
</tem:Operation1>
</soapenv:Body>
</soapenv:Envelope>
"@
},
@{
SOAPAction = "http://tempuri.org/IService1/Operation2";
Body = @"
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/&quot; xmlns:tem="http://tempuri.org/"&gt;
<soapenv:Body>
<tem:Operation1>
<tem:Parameter1>1</tem:Parameter1>
<tem:Parameter2>2</tem:Parameter2>
</tem:Operation1>
</soapenv:Body>
</soapenv:Envelope>
"@
}
)
# Functions
function Write-Log([string]$level, [string]$text) {
$logTime = Get-Date –Format "yyyy-MM-dd HH:mm:ss.fff"
Write-Host "[$level] $text"
if ($logFile -ne $null) {
"$logTime [$level] $text" | Out-File –FilePath $Logfile –Append
}
}
function Cleanup-LogFiles() {
$expiryDate = (Get-Date).AddMinutes(-7)
[bool]$logFileDeleted = $false
$filesToDelete = Get-Childitem $logFolder –Include "*.log" –Recurse | ? {$_.LastWriteTime -le $expiryDate} | % {
Remove-Item $_.FullName | out-null
Write-Log "Information" "File deleted successfully – $_"
$logFileDeleted = $true
}
if (-not $logFileDeleted) {
Write-Log "Information" "No old log file was found in $logFolder"
}
}
function Warmup-WcfService($serviceUrl) {
foreach($webRequest in $webRequests) {
$headers["SOAPAction"] = $webRequest.SOAPAction
Invoke-WcfService $serviceUrl $headers $webRequest.Body
}
}
function Invoke-WcfService($uri, $headers, $body) {
$invocationStartTime = Get-Date
try {
$result = Invoke-WebRequest –Uri $uri –Method Post –UseDefaultCredentials –ContentType "text/xml; charset=utf-8" –Headers $headers –body $body –TimeoutSec $timeoutSec
$invocationDuration = (Get-Date).Subtract($invocationStartTime)
if ($result.StatusCode -eq 200) {
if ($result.Headers.ContainsKey("Content-Length")) {
$contentLength = $result.Headers["Content-Length"]
} else {
$contentLength = "$($result.RawContentLength) (raw)"
}
Write-Log "Information" "WCF operation invoked successfully – Operation: $($webRequest.SOAPAction)($variation), Duration: $($invocationDuration), Length: $contentLength"
} else {
Write-Log "Warning" "WCF operation failed – Operation: $($webRequest.SOAPAction)($variation), Status: $($result.StatusCode) $($result.StatusDescription), Length: $contentLength"
}
} catch {
Write-Log "Critical" "WCF operation failed – Operation: $($webRequest.SOAPAction)($variation), Exception: $_"
}
}
function Invoke-WebPage([string]$uri, [bool]$getImages, [bool]$getScripts) {
try {
$response = Invoke-WebRequest –Uri $uri –UseDefaultCredentials –TimeoutSec $timeoutSec –Proxy "http://localhost" –ProxyUseDefaultCredentials
if ($response.StatusCode -eq 200) {
Write-Log "Information" "Web page invoked successfully – URL: $uri, Images: $($response.Images.Count), Scripts: $($response.Scripts.Count), Links: $($response.Links.Count)"
} else {
Write-Log "Warning" "Web page invocation failed – URL: $uri, Status: $($response.StatusCode) $($response.StatusDescription)"
}
} catch {
Write-Log "Critical" "Web page invocation failed – URL: $uri, Exception: $_"
}
if ($getImages) {
Fetch–Resources $uri ($response.Images | ? { $_.src } | % { $_.src })
}
if ($getScripts) {
Fetch–Resources $uri ($response.Scripts | ? { $_.src } | % { $_.src })
}
}
Function Fetch-Resources($baseUrl, $resources) {
[uri]$uri = $baseUrl
$rootUrl = $uri.Scheme + "://" + $uri.Authority
$counter = 0
foreach ($resUrl in $resources) {
if ($resUrl.StartsWith("http", [System.StringComparison]::OrdinalIgnoreCase)) {
$fetchUrl = $resUrl
} elseif (!$resUrl.StartsWith("/")) {
$fetchUrl = $rootUrl + "/" + $resUrl
} else {
$fetchUrl = $rootUrl + $resUrl
}
# Progress
Write-Progress –Activity "Opening " –Status $fetchUrl –PercentComplete (($counter/$resources.Length)*100)
$counter++
# Execute
try {
$resp = Invoke-WebRequest –UseDefaultCredentials –UseBasicParsing –Uri $fetchUrl –TimeoutSec 120
Write-Log "Information" "Resource has been downloaded successfully – URL: $fetchUrl"
} catch {
Write-Log "Warning" "Resource failed to download – URL: $fetchUrl, Exception: $_"
}
}
Write-Progress –Activity "Completed" –Completed
}
# Execution
Write-Log "Information" "My Web Application Warmup Script – $(Get-Date)"
Write-Log "Information" "Cleaning up old log files…"
Cleanup–LogFiles
Write-Log "Information" "Warming up WCF Services…"
Warmup–WcfServices
Write-Log "Information" "Warming up Front-End…"
Invoke-WebPage "http://domain.com/area1/Page1.aspx" $true $true
Invoke-WebPage "http://domain.com/area1/Page2.aspx" $true $true
Write-Log "Information" "My Web Application Warmup Script Completed."

view raw

Warmup-WebApp.ps1

hosted with ❤ by GitHub

I’m open to any suggestion and feature request. Please let me know if you found it useful or if have got something wrong.

How to create search query content source in SharePoint using PowerShell

They are called Content Sources in the graphical user interface of SharePoint and Search Query Content Source when using PowerShell. A content source is basically pointing to a source of data that can be indexed in the search database. Because the data is indexed, when later users run queries the result is returned almost instantaneous. It is very easy to add a content source by using SharePoint Central Administration, but you might need to create a long list of content sources in a batch for example as part of a disaster recovery process or many other reasons. Here is one way I do it in a PowerShell script.

$contentSourceConfig = @(
    @{
        Name = "Articles"
        StartAddresses = "http://mydomain1/site1/fr,http://mydomain1/site1/nl,http://mydomain1/site1/en"
    },
    @{
        Name = "Technical"
        StartAddresses = "http://mydomain1/site2/fr,http://mydomain1/site2/nl,http://mydomain1/site2/en"
    }
)

foreach($config in $contentSourceConfig) {
    $contentSourceName = $config.Name
    $startAddresses = $config.StartAddresses
    $contentSource = Get-SPEnterpriseSearchCrawlContentSource $contentSourceName -SearchApplication $ssa -ErrorAction SilentlyContinue
    if ($contentSource) { $contentSource.Delete() }
    $contentSource = New-SPEnterpriseSearchCrawlContentSource $contentSourceName -Type "Web" -StartAddresses $startAddresses -SearchApplication $ssa
    Write-Host "Search crawl content source: $contentSourceName created successfully."
}

As you see in the above example I have put all the required configuration in one array of custom objects. If you need to add more content sources you may simply copy-past one and change the properties. This way the configuration is separated and easier to read or change while still it is in the same file with the script so you won’t have to rely on extra XML file(s) to put beside the script.

If you are more comfortable by separating the configuration from the script. You can cut the first part and past it to another script file or use the XML approach.

For more details about the cmdlets I’ve used I suggest you have a look on the following MSDN articles:

First step in developing apps for SharePoint Online – Enabling F5

It’s Just a quick post to share one of my first experiences with SharePoint online and that’s how to enable F5 experience in Visual Studio. When you start writing your first App for SharePoint Online you might get the same error as me, saying that “Sideloading of apps is not enabled on this site”

Sideloading of apps is not enabled on this site

Sideloading of apps is not enabled on this site

It’s because the “Developer” feature is not activated by default and you need to activate it manually, but since it’s an online version of SharePoint things are a little bit different that you local development environment (a good article on how to prepare your local development environment) and you need to use CSOM (Client Side Object Model). I have just translated the c# equivalent to a few lines of PowerShell code so you can type it into your SharePoint Online Management Shell (if you have not installed it, you can download it from here). Here is the code:

<br />$siteurl = "https://yourwebsite.sharepoint.com"<br />$username = "yourusername"<br />$password = ConvertTo-SecureString -String 'yourpassword!' -AsPlainText -Force<br />[Microsoft.SharePoint.Client.ClientContext] $context = New-Object Microsoft.SharePoint.Client.ClientContext($siteurl)<br />[Microsoft.SharePoint.Client.SharePointOnlineCredentials] $credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $password)<br />$context.Credentials = $credentials<br />$site = $cc.Site<br />$sideLoadingGuid = New-Object System.Guid "AE3A1339-61F5-4f8f-81A7-ABD2DA956A7D"<br />$site.Features.Add($sideLoadingGuid, $true, [Microsoft.SharePoint.Client.FeatureDefinitionScope]::None)<br />$context.ExecuteQuery();<br />

Add / Remove event receivers to list / content types in SharePoint with PowerShell

Although it’s simple but it’s a little bit tricky. Adding / removing an event receiver to a list and content type is the same you just need to use the appropriate collection is an SPWeb object.

Here is how to add / remove an event receiver to / from a list:

# The following code registers an event receiver for when an item is being added to a list
$web = Get-SPWeb "http://yourwebsiteurl"
$list = $web.Lists["your list name"]
$list.EventReceivers.Add("ItemAdding", "Full Assembly Name", "Full.Class.Name")

Removing an event receiver is as easy as:

# Be careful about the index value for when you have more than one event receiver!
$list.EventReceiver[0].Delete()

As you can see below adding an event receiver to a content type is very similar:

# The following code registers an event receiver for when an item is being added to a list
$web = Get-SPWeb "http://yourwebsiteurl"
$ctype = $web.ContentTypes["your content type name"]
$ctype.EventReceivers.Add("ItemAdding", "Full Assembly Name", "Full.Class.Name")

And so is deleting it:

# Be careful about the index value for when you have more than one event receiver!
$ctype.EventReceiver[0].Delete()

Now more important is the tricky part. If you have used this content type to create other / list instances then you should also call the following command to propagate the change:

$ctype.Update($true)