RubberDuckDev

Microsoft Authenticator - "Unexpected error" when scanning QR code

Mon, 08 Jun 2026 10:00:00 GMT

Introduction

While onboarding a brand new user to a tenant, the Microsoft Authenticator app refused to register the account, even though other users on the same tenant were registering without any issue. This post captures the symptoms and the fix so future-me (and you) don’t have to spend a week on a support ticket to find it.

The setup and issue

The flow that failed was a textbook first-time MFA setup:

New user is created in Microsoft Entra ID.
User signs in to https://login.microsoftonline.com with the temporary password.
User sets a new password.
User is prompted to set up MFA and chooses Work or school account in Microsoft Authenticator.
User scans the QR code.

At step 5, the Authenticator app fails with:

Unable to add account.
Unexpected error. Please contact your local IT administrator to resolve the problem.

Things worth noting:

Other (existing and newly created) users could add their accounts to Authenticator on different devices without any problem.
The same Authenticator app on the affected device already had other work accounts registered and was working fine.
The tenant configuration, Conditional Access, and user state all looked healthy.

So the failure was clearly local to the Authenticator app’s state on that one device, not a tenant or account issue.

The solution

The fix turned out to be a single, low-key option inside the Authenticator app:

Authenticator app → Settings → Device Registration → Reset Device Notifications.

After triggering Reset Device Notifications, the QR code scan completed and the account was added on the first try.

Why it works

When you add a work or school account, the Authenticator app must complete two things behind the scenes:

Register the app/device with Microsoft Entra ID.
Establish a push notification channel (via FCM on Android / APNs on iOS) so MFA prompts can be delivered.

If the local notification registration state is stale or inconsistent - for example after OS updates, app updates, restoring from backup, or partially failed previous registrations - the registration handshake can fail and surface as the generic “Unexpected error”.

Reset Device Notifications clears that local state and forces the app to re-establish a fresh notification registration, which unblocks the new account add.

Conclusion

If you ever hit “Unable to add account. Unexpected error.” in Microsoft Authenticator and the tenant side looks healthy, try Reset Device Notifications before going down the rabbit hole of factory resets or new devices. Hope this saves you some time.

When git reset --hard Isn’t Enough

Fri, 05 Jun 2026 00:00:00 GMT

Introduction

I ran into a small but frustrating issue recently. Git was telling me my branch was ahead of origin by a couple of commits, even though I just wanted to throw everything away and go back to the remote version.

My first instinct was the usual:

git reset --hard

But nothing changed. The “2 unpushed commits” were still there.

What’s actually happening

git reset --hard on its own only resets your working directory to your current HEAD. If your local branch already includes those extra commits, Git considers that the correct state. So it doesn’t remove anything.

In other words, you’re resetting to the wrong place.

The fix

What worked straight away was resetting to the remote branch instead:

git fetch origin
git reset --hard origin/branch-name

Why this works

git fetch origin updates your view of the remote
git reset --hard origin/branch-name moves your local branch pointer back to exactly where the remote branch is

This effectively discards any local commits that haven’t been pushed.

A quick note

This is a destructive operation. Any unpushed commits will be lost, so it’s worth double-checking that you really don’t need them.

Takeaway

If you’re trying to get your branch back in sync with the remote and git reset --hard doesn’t help, it’s probably because you’re resetting to your own HEAD. Point it at origin/branch-name instead, and the issue goes away.

Why Try-Catch Doesn't Work with Pulumi

Fri, 10 Oct 2025 16:30:00 GMT

When working with Pulumi, especially coming from traditional .NET development, you might expect that wrapping resource creation in a try-catch block would handle deployment errors gracefully. However, as many developers discover, this approach doesn’t work as expected. In this post, we’ll explore why this happens and how to properly handle errors in Pulumi applications.

The Problem: When Try-Catch Fails

Consider this seemingly straightforward Pulumi C# program:

using Pulumi.AzureNative.Storage;
using Pulumi.AzureNative.Storage.Inputs;
using System;

return await Pulumi.Deployment.RunAsync(() =>
{
    try
    {
        // Create an Azure resource (Storage Account)
        var storageAccount = new StorageAccount("sa", new StorageAccountArgs
        {
            ResourceGroupName = "non-existent-rg", // Intentionally incorrect
            Sku = new SkuArgs
            {
                Name = SkuName.Standard_LRS
            },
            Kind = Kind.StorageV2
        });
    } 
    catch (Exception ex)
    {
        // This should catch errors, right? Wrong!
        Console.WriteLine($"An error occurred: {ex.Message}");
    }
});

Running pulumi preview should work fine and it should report the expected values:

+ pulumi:pulumi:Stack: (create)
    [urn=urn:pulumi:testtrycatch::pulumi-try-catch-fail::pulumi:pulumi:Stack::pulumi-try-catch-fail-testtrycatch]
    + azure-native:storage:StorageAccount: (create)
        [urn=urn:pulumi:testtrycatch::pulumi-try-catch-fail::azure-native:storage:StorageAccount::sa]
        [provider=urn:pulumi:testtrycatch::pulumi-try-catch-fail::pulumi:providers:azure-native::default_3_8_0::04da6b54-80e4-46f7-96ec-b56ff0331ba9]
        accountName         : "sa9203a871"
        azureApiVersion     : "2024-01-01"
        kind                : "StorageV2"
        location            : "westeurope"
        resourceGroupName   : "non-existent-rg"
        sku                 : {
            name: "Standard_LRS"
        }
Resources:
    + 2 to create

When running pulumi up on this code, you might expect the try-catch to handle any deployment errors gracefully. Instead, you get an error like this:

error: cannot check existence of resource '/subscriptions/.../resourceGroups/non-existent-rg/providers/Microsoft.Storage/storageAccounts/sa79a4d803': status code 403, {"error":{"code":"AuthorizationFailed","message":"The client '...' does not have authorization to perform action 'Microsoft.Storage/storageAccounts/read' over scope '...' or the scope is invalid."}}
error: update failed

Three questions arise:

Why doesn’t the try-catch hide the error?
Why doesn’t the error mention that the resource group doesn’t exist?
Why isn’t the Console.WriteLine message displayed?

Understanding Pulumi’s Execution Model

The Asynchronous, Declarative Nature

The fundamental issue lies in understanding when Pulumi actually creates resources. When you write:

var storageAccount = new StorageAccount("sa", new StorageAccountArgs { ... });

This line does not immediately create the Azure resource. Instead, it:

Creates a Pulumi resource object in memory
Registers it with the Pulumi engine for later deployment
Returns immediately without making any Azure API calls

The actual Azure API calls happen later during the deployment phase, which occurs after your C# code has completely finished executing.

Why Try-Catch Doesn’t Work

The try-catch block in your code only catches exceptions that occur during the registration phase (steps 1-2 above), not during the deployment phase where the actual Azure API calls are made.

try
{
    // This only registers the resource - no Azure API calls yet
    var storageAccount = new StorageAccount("sa", args);
    // Execution reaches here successfully
} 
catch (Exception ex)
{
    // This will never execute for deployment errors
    Console.WriteLine($"An error occurred: {ex.Message}");
}
// C# code completes here
// THEN Pulumi makes the actual Azure API calls

Why the Error Message Doesn’t Mention Resource Group

The error you’re seeing is an authorization error (HTTP 403), not a “resource not found” error (HTTP 404). Here’s the sequence:

Azure checks permissions first: Before checking if resources exist, Azure validates whether you have the required permissions
Permission denied: Since the resource doesn’t exist, it assumes user doesn’t have Microsoft.Storage/storageAccounts/read permission on that scope, Azure returns a 403 error
Early termination: Azure’s RBAC (Role-Based Access Control) evaluation happens before resource existence validation

So we get a 403 as permissions could not be validated.

Proper Error Handling Strategies in Pulumi

1. Validate Dependencies First

The most straightforward approach is to ensure your dependencies exist:

return await Pulumi.Deployment.RunAsync(() =>
{
    // Create the resource group first
    var resourceGroup = new ResourceGroup("rg", new ResourceGroupArgs
    {
        Location = "WestEurope"
    });

    // Use the created resource group
    var storageAccount = new StorageAccount("sa", new StorageAccountArgs
    {
        ResourceGroupName = resourceGroup.Name, // Reference the created RG
        Sku = new SkuArgs { Name = SkuName.Standard_LRS },
        Kind = Kind.StorageV2
    });
});

2. Use Pulumi’s Apply Methods for Output Handling

You can handle errors in resource outputs using the Apply method:

var storageAccount = new StorageAccount("sa", new StorageAccountArgs
{
    ResourceGroupName = "my-resource-group",
    Sku = new SkuArgs { Name = SkuName.Standard_LRS },
    Kind = Kind.StorageV2
});

// Handle the output with potential error scenarios
var storageAccountId = storageAccount.Id.Apply(id => 
{
    Console.WriteLine($"Storage account created with ID: {id}");
    return id;
});

3. Use Stack Transformations for Advanced Scenarios

For more complex error handling or resource modification scenarios:

return await Pulumi.Deployment.RunAsync(() =>
{
    // Set up stack transformation for resource validation
    Pulumi.Deployment.Instance.RegisterStackTransformation(args =>
    {
        // You can validate or modify resources here before deployment
        if (args.Resource is StorageAccount sa)
        {
            // Add validation logic
            Console.WriteLine($"Validating storage account: {args.Name}");
        }
        return args;
    });

    var storageAccount = new StorageAccount("sa", new StorageAccountArgs
    {
        ResourceGroupName = "my-resource-group",
        Sku = new SkuArgs { Name = SkuName.Standard_LRS },
        Kind = Kind.StorageV2
    });
});

4. Use Pulumi’s Built-in Resource Options

Pulumi provides several resource options for handling various scenarios:

var storageAccount = new StorageAccount("sa", new StorageAccountArgs
{
    ResourceGroupName = "my-resource-group",
    Sku = new SkuArgs { Name = SkuName.Standard_LRS },
    Kind = Kind.StorageV2
}, new CustomResourceOptions
{
    // Ignore changes to specific properties
    IgnoreChanges = { "tags" },
    
    // Protect the resource from deletion
    Protect = true,
    
    // Set explicit dependencies
    DependsOn = { /* other resources */ }
});

Best Practices for Error Prevention

Use Resource References Instead of Hard-coded Names

// Bad: Hard-coded resource group name
ResourceGroupName = "my-resource-group"

// Good: Reference to created resource
ResourceGroupName = resourceGroup.Name

Conclusion

Understanding Pulumi’s asynchronous, declarative execution model is crucial for effective error handling. Traditional try-catch blocks work for registration-time errors but not for deployment-time errors.

Additional Resources

For more insights on Infrastructure as Code and cloud development best practices, follow my blog or connect with me on LinkedIn.

Git Branch Naming Conflicts in Azure DevOps

Wed, 01 Oct 2025 17:30:00 GMT

Special thanks to my fantastic colleague Fatih Kucukkara for helping figure these details out.

🧩 What They Are and How to Fix Them

If you’ve ever pushed a branch to Azure DevOps and been greeted with a cryptic error like this:

[remote rejected] bug/456789-fix-something -> bug/456789-fix-something (Name conflicts with refs/heads/Bug/987654-fix-another-thing)

You’re not alone. This error can be frustrating, especially when everything looks fine locally. Let’s break down what’s happening and how to resolve it.

🔍 Understanding the Error

This error means that Azure DevOps is rejecting your push because the branch name you’re trying to use conflicts with an existing branch name—but not in the way you might expect.

The culprit? Case sensitivity.

While Git is case-sensitive on most systems, Azure DevOps treats branch names as case-insensitive. So if you have:

bug/456789-fix-something
Bug/987654-fix-another-thing

Azure DevOps sees these as potentially conflicting because it doesn’t distinguish between bug/ and Bug/.

🛠️ How to Fix It

Here are a few ways to resolve this issue:

✅ 1. Rename Your Branch Locally

Avoid the conflict by renaming your branch to something unique:

git branch -m bug/456789-fix-something bugfix/456789-fix-something
git push origin bugfix/456789-fix-something

This sidesteps the naming conflict entirely.

✅ 2. Delete the Conflicting Remote Branch (If Safe)

If the conflicting branch is no longer needed and you’re sure it’s safe to remove:

git push origin --delete Bug/987654-fix-another-thing

Then retry your push.

✅ 3. Adopt a Clear Naming Convention

To prevent future conflicts, consider using a consistent and distinct naming convention for branches—such as feature/, bugfix/, hotfix/, etc.—and avoid mixing cases.

🧠 Final Thoughts

This issue is a great reminder that case sensitivity can behave differently across systems, and that naming conventions matter—especially in collaborative environments like Azure DevOps.

Multi-Agent AI App with Microsoft Semantic Kernel

Sat, 27 Sep 2025 21:05:52 GMT

Learn how to build sophisticated AI applications using Microsoft Semantic Kernel with multi-agent architecture. Covers Azure OpenAI, Google AI Studio, Docker deployment, and innovative use cases for specialized AI agents.

Building Multi-Agent AI Applications with Microsoft Semantic Kernel

The rise of AI-powered content generation has opened new possibilities for automating complex workflows. By combining Microsoft Semantic Kernel with a multi-agent architecture, developers can create sophisticated applications that break down complex tasks into specialized, manageable components.

What is Microsoft Semantic Kernel?

Microsoft Semantic Kernel is an open-source SDK that enables developers to integrate AI services into their applications seamlessly. It provides a unified interface for working with various AI models, whether from Azure OpenAI, Google AI, or local deployments. The framework emphasizes function-based AI interactions, allowing developers to define specific capabilities as kernel functions. It supports multiple languages, in this post we will use C#.

The dotnet package we need to use Semantic Kernel in dotnet:

dotnet package add Microsoft.SemanticKernel

A sample Kernel Function example:

[KernelFunction, Description("Write blog post content based on research outline")]
public async Task<string> WriteContentAsync(string outline, string tone = "Professional")
{
    var prompt = $"You are an expert content writer...";
    var result = await _kernel.InvokePromptAsync(prompt);
    return result.ToString();
}

Multi-Agent Architecture: Divide and Conquer

Traditional AI applications often try to handle everything in a single prompt or interaction. Multi-agent systems take a different approach by creating specialized agents, each responsible for a specific aspect of the workflow. This architecture offers several advantages:

Specialized Expertise: Each agent focuses on one task—research, writing, editing, SEO optimization, or markdown formatting. This specialization leads to higher quality outputs as each agent can be fine-tuned for its specific role.

Maintainability: Changes to one agent don’t affect others, making the system easier to maintain and extend.

Pipeline Processing: Complex tasks are broken into manageable steps, with each agent building upon the previous one’s output.

A typical multi-agent blog generation pipeline might flow like this: Research Agent → Content Writer → Editor → Markdown Linter → SEO Optimizer

Flexible AI Provider Support

Modern applications need flexibility in choosing AI providers based on cost, performance, and availability requirements.

Azure OpenAI Integration

Azure OpenAI provides enterprise-grade AI services with enhanced security and compliance features. Integration is straightforward:

dotnet package add Microsoft.SemanticKernel.Connectors.OpenAI

builder.AddAzureOpenAIChatCompletion(
    deploymentName: "gpt-4o-mini",
    endpoint: "https://your-deployment.openai.azure.com/",
    apiKey: "your-api-key"); // Important - do not use in plain text, use a secured mechanism

Google AI Studio: Free Tier Benefits

Google AI Studio offers competitive pricing with a generous free tier, making it an excellent choice for development and small-scale applications. According to Google’s pricing documentation, developers can access Gemini models with substantial free monthly quotas before incurring charges. This makes it particularly attractive for:

Prototyping and development
Educational projects
Small-scale applications
Cost-conscious implementations

dotnet package add Microsoft.SemanticKernel.Connectors.Google --prerelease

builder.AddGoogleAIGeminiChatCompletion(
                modelId: "gemini-2.5-pro,
                apiKey: "your-api-key"); // Important - do not use in plain text, use a secured mechanism

Local Development with Docker

For development environments or when working with sensitive data, running models locally provides complete control and privacy. Docker Model Runner (DMR) makes this surprisingly accessible:

# Pull and run a local AI model
docker model pull ai/gemma3
docker model run ai/gemma3

// Create HttpClient with longer timeout as local run can take more than 100ms
var httpClient = new HttpClient
{
    // Increase timeout as local runs take long
    Timeout = TimeSpan.FromMinutes(10)
};

builder.AddOpenAIChatCompletion(
                modelId: "ai/gemma3", // note that the id has to match exactly what we pulled
                apiKey: "not-required",
                endpoint: new Uri("http://localhost:12434/engines/v1"), // the port number is set in docker-desktop -> Settings -> AI -> Enable Docker Model Runner -> Enable host-side TCP support -> Port
                httpClient: httpClient);

Local deployment offers several benefits:

Privacy: Data never leaves your environment
Cost Control: No per-token charges
Offline Capability: Works without internet connectivity
Customization: Full control over model parameters

Implementation Benefits

This multi-agent approach delivers tangible benefits:

Quality Through Specialization: Each agent excels at its specific task rather than being a generalist.

Scalability: Individual agents can be optimized, replaced, or scaled independently.

Debugging: Issues can be traced to specific agents, making troubleshooting more straightforward.

Flexibility: Easy to swap AI providers or add new agents without restructuring the entire application.

Future Enhancement Ideas

The multi-agent architecture opens doors for exciting improvements:

Intelligent Routing: Implement dynamic agent selection based on content type, complexity, or quality requirements.

Quality Scoring: Add agents that evaluate output quality and trigger rewrites when necessary.

Multi-Modal Capabilities: Integrate image generation agents for creating accompanying visuals, diagrams, or social media graphics.

Real-Time Fact Checking: Incorporate agents that verify claims against current data sources or knowledge bases.

Personalization Engine: Develop agents that adapt content style and complexity based on reader preferences or analytics.

Translation Pipeline: Add multilingual agents for automatic content localization.

Performance Analytics: Implement agents that analyze content performance and suggest optimizations.

Beyond Content: Creative Multi-Agent Applications

The multi-agent paradigm extends far beyond content generation into innovative domains:

Digital Archaeology: Deploy specialized agents to reconstruct lost digital assets—one agent analyzes fragmented code repositories, another reconstructs missing documentation, while a third validates historical API behaviors through pattern recognition.

Synthetic Dataset Generation: Create agents that collaborate to generate realistic training data. A demographic agent creates diverse user profiles, a behavioral agent simulates realistic interaction patterns, and a validation agent ensures statistical authenticity without privacy violations.

Adaptive Learning Orchestration: Build educational systems where agents specialize in different learning modalities—one tracks cognitive load through interaction patterns, another adjusts difficulty curves in real-time, and a third generates personalized analogies based on the learner’s background knowledge.

Regulatory Compliance Automation: Deploy agents across different regulatory frameworks where each agent becomes an expert in specific compliance domains (GDPR, SOX, HIPAA), automatically scanning systems and suggesting remediation strategies.

Creative Ideation Networks: Implement agents that approach problem-solving from different creative methodologies—one uses lateral thinking techniques, another applies biomimicry principles, while a third challenges assumptions through contrarian analysis.

Automated Research Hypothesis Generation: Deploy agents that scan scientific literature, identify knowledge gaps, and propose novel research directions by combining insights across traditionally separate domains.

These applications demonstrate how multi-agent systems can tackle complex, nuanced problems that require diverse expertise and collaborative intelligence—opening possibilities we’re only beginning to explore.

Conclusion

Multi-agent AI systems represent a maturation of AI application development, moving beyond monolithic AI interactions toward specialized, maintainable, and scalable architectures. By leveraging tools like Microsoft Semantic Kernel and embracing the multi-agent paradigm, developers can create applications that are not only more capable but also more reliable and easier to maintain.

Whether you’re building content generation tools, data analysis pipelines, or complex decision support systems, the multi-agent approach offers a pathway to more sophisticated and maintainable AI applications.

Please note, I have this old branch using Semantic Kernel for reference if anyone is interested. The repo has now moved to use Microsoft Agent Framework.

Update Docker Desktop CLI - Fast & Easy

Sat, 27 Sep 2025 11:05:52 GMT

This guide demonstrates how to efficiently update Docker Desktop directly from the command line using the Docker Desktop CLI. It's a faster, more reliable alternative to the GUI, ideal for developers and DevOps teams. Learn step-by-step instructions and scripting tips for automated updates.

Level Up Your Docker Workflow

Are you tired of the tedious process of updating Docker Desktop? Or maybe your corporate policies is blocking you from updating docker from the UI? Fortunately, there’s a smarter, faster way. The Docker Desktop CLI offers a powerful and reliable solution for updating Docker Desktop directly from the command line, significantly boosting your productivity and streamlining your workflows.

What is the Docker Desktop CLI?

The Docker Desktop CLI (Command-Line Interface) is a tool that provides a way to manage key features of Docker Desktop directly from your terminal. Introduced in version 4.39, it’s surprisingly easy to use, refer to the official documentation: https://docs.docker.com/desktop/features/desktop-cli/

Step-by-Step Update Process

Let’s walk through the simple process of updating Docker Desktop using the CLI:

Step 1: Verify Version

First, check your current Docker Desktop version:

docker version

This command will display information about your Docker client and server versions. Ensure your Docker Desktop version is 4.39 or later to utilize the CLI’s update functionality.

Step 2: Execute the Update Command

Now, initiate the update process:

docker desktop update

This command tells Docker Desktop to check for available updates and apply them. Behind the scenes, the CLI meticulously checks for new versions, downloads the necessary files, and restarts Docker Desktop if required – all without the need to manually interact with the GUI.

Step 3: Post-Update Verification

After the update completes, confirm the new version:

docker version

You should now see the updated Docker version displayed.

Linux Update Considerations

It’s important to note that Docker Desktop isn’t the primary tool for managing updates on Linux systems. Instead, utilize your distribution’s package manager (apt, yum, dnf) to ensure a consistent and supported Docker installation. For example, on Debian/Ubuntu:

sudo apt update && sudo apt upgrade

Conclusion & Call to Action

The Docker Desktop CLI is a valuable tool that can significantly improve your Docker workflow efficiency. By automating updates and providing greater control, it’s a must-have for developers, DevOps engineers, and students alike. Start leveraging the CLI today and experience the difference! Explore the official documentation https://docs.docker.com/desktop/features/desktop-cli/ to learn more and unlock the full potential of your Docker Desktop environment.

Offline AI - Integrating Local LLMs with VSCode

Thu, 15 May 2025 14:00:00 GMT

Introduction

In the age of AI, developers are increasingly looking for ways to integrate powerful language models into their workflows. While cloud-based solutions like OpenAI’s GPT models are popular, there’s a growing demand for offline and private alternatives. This guide walks you through integrating a locally running LLM (DeepSeek R1 Distill LLaMA) with the Continue extension in Visual Studio Code.

Why Offline Mode is Useful

Running AI models locally offers several advantages:

Privacy: Sensitive codebases and data never leave your machine, ensuring compliance with privacy policies and regulations.
Speed: Local models eliminate latency caused by network requests, providing near-instant responses.
Cost Efficiency: Avoid recurring API costs by leveraging your local hardware.
Customization: Fine-tune models to your specific needs without relying on external providers.
Offline Access: Work uninterrupted, even without an internet connection.

For developers who value control and independence, offline AI development is a game-changer.

The full details is on github at continue-local-llm-integration.

🐳 Step 1: Run the LLM in Docker

To get started, you’ll need to run the DeepSeek R1 Distill LLaMA model locally using Docker. This model is OpenAI-compatible and provides a robust foundation for offline AI development.

Resources:

Steps:

Pull the DeepSeek model from Docker Hub.
Start the container and ensure it exposes an endpoint like:
```
http://localhost:12434/engines/v1/chat/completions
```
Test the setup using the provided test.ps1 script in this repository.

🧠 Step 2: Install the Continue Extension

The Continue extension for Visual Studio Code is a powerful tool that brings AI-assisted coding to your IDE. It supports OpenAI-compatible APIs, making it an ideal choice for integrating local LLMs.

Installation:

Open Visual Studio Code.
Navigate to the Extensions Marketplace.
Search for “Continue” and install the extension.

⚙️ Step 3: Configure Continue

To connect the Continue extension to your locally running LLM, you’ll need to update its configuration.

Locate the sample-config\config.yaml file in this repository.
Copy the file to %userprofile%/.continue on your machine.
Update the configuration to point to your local LLM endpoint:
```
api_url: http://localhost:12434/engines/v1/chat/completions
```

⚽ Step 4: Start Coding with AI

Once everything is set up, you can start leveraging the power of local AI models directly in your IDE. The Continue extension provides features like code completion, refactoring suggestions, and more.

Conclusion

By running models offline, you gain privacy, speed, and control over your development environment. Having said so, running models locally takes up a lot of resource, at least on my machine. Hopefully the experience is smoother on yours.

Credits

Banner image from BoliviaInteligente

Avoid Storing Secrets in PowerShell's Command History

Mon, 17 Mar 2025 17:30:00 GMT

Introduction

If you’re using PowerShell scripts to manage secrets like API tokens, passwords, or other sensitive data, you may think you’re safe by using environment variables or script parameters like $PAT or $SomePassword to avoid hardcoding secrets in your scripts. However, there’s a common pitfall that many PowerShell users overlook: the PowerShell command history.

PowerShell saves every command you execute in a file located at:

%userprofile%\AppData\Roaming\Microsoft\Windows\PowerShell\PSReadline\ConsoleHost_history.txt

This file stores all your PowerShell commands, including those that might contain sensitive information, like manually setting a parameter for a script:

$PAT = 'mySuperSecretToken'

Even though you’re not committing this secret to Git, it’s still saved in plain text in your command history, which means it’s just as vulnerable if someone accesses your machine. In this blog post, we’ll look at how to prevent this from happening.

Why is this a Security Concern?

If anyone gains access to your machine and navigates to the ConsoleHost_history.txt file, they can see all the commands you’ve executed, including any credentials or secrets you’ve input manually.

Here’s an example of what could happen:

cd C:\Projects\MyProject
$SomePassword = 'superSecurePassword123!'
git push origin main

A simple look at your PowerShell history could reveal sensitive data like passwords, API tokens, or private keys.

How to Prevent Secrets from Appearing in PowerShell History

1. Clear History After Entering Sensitive Information

One simple way to prevent secrets from being stored is to clear the PowerShell history immediately after running a sensitive command:

Clear-History

This will remove all commands from the current session’s memory. However, it doesn’t retroactively remove sensitive commands already written to ConsoleHost_history.txt. You should also regularly clear the file manually:

Remove-Item "$env:APPDATA\Microsoft\Windows\PowerShell\PSReadLine\ConsoleHost_history.txt"

2. Use Secure String for Secrets

Rather than passing secrets as plain text, you can use PowerShell’s SecureString to store sensitive data. While this won’t stop it from being written to the history, it helps protect it when stored or transmitted:

$SecurePassword = Read-Host "Enter Password" -AsSecureString

This method prompts the user to enter the secret interactively, and the input won’t be logged in the command history.

3. Disable Command History

If you often deal with sensitive information in your PowerShell sessions, it might be best to disable the history feature entirely. You can do this by editing the PowerShell profile file:

Open the PowerShell profile for editing:
```
notepad $PROFILE
```
Add the following line to disable PSReadLine’s history recording:
```
Set-PSReadlineOption -HistorySaveStyle SaveNothing
```
Save the file and restart PowerShell. Now, PowerShell won’t save any commands to history.

4. Use Environment Variables for Sensitive Data

To further reduce risk, you can store sensitive information in environment variables. These won’t be saved to history, as they are set outside the scope of PowerShell:

$PAT = $env:MY_PERSONAL_ACCESS_TOKEN

Just make sure that environment variables containing secrets are not stored in plaintext in scripts or configuration files.

Best Practices for Handling Secrets

Avoid typing secrets directly into the PowerShell prompt. Always use Read-Host with the -AsSecureString flag to securely capture input.
Clear the command history if you’ve entered sensitive information manually during a session.
Store secrets in a dedicated secret management system (like Azure Key Vault, HashiCorp Vault, or AWS Secrets Manager), and retrieve them programmatically instead of hardcoding or passing them manually.
Disable history recording when working in environments where sensitive data will be routinely entered.
Regularly audit your command history to ensure no sensitive data is accidentally exposed.

Conclusion

The PowerShell command history can be a double-edged sword: it’s helpful for recalling past commands, but it can expose sensitive information if not managed carefully. If you’re working with secrets in PowerShell, take steps to prevent them from ending up in the command history file by using secure input methods, clearing history, or disabling it altogether. By following the best practices outlined above, you can better safeguard sensitive data in your PowerShell environment.

Stay safe and keep your secrets secure!

Comparing Two Branches in Azure DevOps

Sat, 25 Jan 2025 00:00:00 GMT

This is a nice trick, which I was not aware of, even though I have been using Azure DevOps for many years. So thought will write a post about it so hopefully someone else benefits from this.

When working with multiple branches in a repository, it’s useful to compare them to understand the differences and ensure that changes are correctly merged. Azure DevOps provides a straightforward way to compare branches and review the changes.

Step-by-Step Guide

Navigate to Your Repository
- Open your Azure DevOps project.
- Go to the “Repos” section from the left-hand menu.
- Select the repository you want to work with.
Select the Branches

Click on the “Branches” tab to view all the branches in your repository.
Identify the two branches you want to compare.
Initiate the Comparison
- Click on the branch you want to compare from.
- In the branch details view, click on the “Compare” button.
Choose the Target Branch
- In the comparison view, select the target branch you want to compare against from the dropdown menu.
- Azure DevOps will display the differences between the two branches, including commits, file changes, and code differences.
Review the Changes
- Use the comparison view to review the changes between the branches.
- You can see the list of commits, files that have been changed, added, or deleted, and the specific code differences.

Conclusion

Comparing branches in Azure DevOps helps in maintaining code quality and collaboration within your team. By following these steps, you can easily compare branches, review changes, and make informed decisions about merging.

I hope this helps! If you have any questions or need further assistance, feel free to ask.

Mocking Azure.Data.AppConfiguration.ConfigurationClient using Moq

Mon, 02 Sep 2024 23:30:00 GMT

Introduction

When working with Azure’s App Configuration service in .NET, it’s common to use the Azure.Data.AppConfiguration.ConfigurationClient class to interact with our configuration settings. In a unit test scenario, however, interacting directly with the Azure service is not ideal. Instead, we can mock the ConfigurationClient using a library like Moq to simulate its behaviour, allowing us to test our code in isolation.

In this blog post, we’ll explore how to mock the ConfigurationClient and simulate the paging behaviour of its GetConfigurationSettings method using Moq. This allows us to unit test our code effectively without making actual calls to Azure.

Setup

Before diving into the code, ensure we have the following packages installed in our project:

Azure.Data.AppConfiguration
Moq
Azure.Core

we can install them via NuGet:

dotnet add package Azure.Data.AppConfiguration
dotnet add package Moq
dotnet add package Azure.Core

Example Code

Let’s look at an example where we mock the GetConfigurationSettings method of ConfigurationClient. We’ll create a mock response that returns a list of ConfigurationSetting objects and simulate the paging behaviour.

// Arrange
var endpoint = "https://example.azure-appconfig.net";
var filter = "KeyFilter";
var label = string.Empty;
var expectedValues = new List<string> { "Value1", "Value2", "Value3" };

// Create a mock of the ConfigurationClient
var configurationClient = new Mock<ConfigurationClient>();

// Simulate a page of ConfigurationSetting objects
var configurationSettingPage = Page<ConfigurationSetting>.FromValues(
    new List<ConfigurationSetting>
 {
        new ConfigurationSetting("Key1", "Value1", label),
        new ConfigurationSetting("Key2", "Value2", label),
        new ConfigurationSetting("Key3", "Value3", label)
 },
    continuationToken: null,
    response: Mock.Of<Response>()
);

// Create a Pageable response that contains the page
var response = Pageable<ConfigurationSetting>.FromPages(
    new List<Page<ConfigurationSetting>> { configurationSettingPage }
);

// Set up the mock to return the simulated pageable response when GetConfigurationSettings is called
configurationClient
 .Setup(x => x.GetConfigurationSettings(It.IsAny<SettingSelector>(), It.IsAny<CancellationToken>()))
 .Returns(response);

Explanation

Let’s break down the code step by step:

Arrange the Test Setup: We start by defining the endpoint, filter, and label typically used in the GetConfigurationSettings method call. We also define the expected values we want our mock to return.
Create a Mock ConfigurationClient: We use Moq to create a mock instance of the ConfigurationClient.
Simulate a Page of Configuration Settings: The Page<ConfigurationSetting> class represents a single page of results. We use the FromValues method to create a page containing three ConfigurationSetting objects. Each setting has a key and value pair.
Create a Pageable Response: The Pageable<T> class represents a sequence of pages. We use FromPages to create a pageable response containing the page we just created. This simulates the response we’d get from the real ConfigurationClient.
Set up the Mock Behavior: Finally, we set up the mock ConfigurationClient to return our pageable response when the GetConfigurationSettings method is called with any SettingSelector and CancellationToken.

Conclusion

Mocking Azure services like ConfigurationClient allows us to isolate our unit tests from external dependencies, ensuring that our tests are fast, reliable, and repeatable. By using Moq and Azure SDK’s built-in paging support, we can simulate complex behaviours like pagination and ensure our code handles these scenarios correctly.

For more detailed information on mocking with the Azure SDK and Moq, refer to the official Microsoft documentation.

Managing Identity and Role Assignments in Pulumi with C#

Mon, 19 Aug 2024 10:30:00 GMT

TLDR

If you need to rename role assignment, rename identity too. Otherwise deployment will fail with role already exists error as Pulumi will try to create new role before deleting old one. This if DeleteBeforeReplace is not already in state. If it is in state, role exists error may not happen.

If you need to rename identity name, rename Cosmos RBAC role assignments (e.g. DocumentDB.SqlResourceSqlRoleAssignment). Otherwise issues like “Updating SQL Role Assignment Principal ID is not permitted. You may only update the associated Role Definition.” are likely to happen.

Introduction

When working with infrastructure as code (IaC) tools like Pulumi, particularly when provisioning cloud resources in Azure, handling resource identities and their associated role assignments is critical. A common scenario arises when you need to rename either a resource’s identity or its role assignment. If not handled correctly, this can lead to errors that can disrupt your deployment process. This post will guide you through handling these situations using C# in Pulumi.

Problem Overview

In Azure, resources like databases, storage accounts, or virtual machines often have associated identities and role assignments. When renaming these, Pulumi may encounter issues if it tries to create new resources before deleting the old ones. This can lead to errors such as:

Role assignment already exists: Occurs if Pulumi tries to create a new role assignment before deleting the old one.
Updating SQL Role Assignment Principal ID is not permitted: Occurs if Pulumi attempts to update certain properties of an existing Cosmos RBAC role assignment directly, which Azure does not allow.

Key Concepts

Before diving into code examples, let’s clarify a few key concepts:

DeleteBeforeReplace: This is a Pulumi option that controls whether Pulumi should delete an existing resource before creating its replacement. When renaming resources, setting this option can help avoid conflicts.
Role Assignments: These define the permissions for an identity over a particular scope (e.g., a resource group or a database). Azure does not allow modifying the PrincipalId (the identity) of an existing role assignment directly; instead, you must delete and recreate the assignment.
Identity: This can be a managed identity in Azure that is assigned to a resource. It is often tied to role assignments, which give it specific permissions.

Handling Renaming Scenarios

1. Renaming Role Assignment and Identity Simultaneously

If you need to rename both role assignment and its associated identity, you must be cautious. Pulumi’s default behavior might try to create the new role assignment before deleting the old one, leading to a “role already exists” error. To avoid this:

Rename the role assignment: Make sure to rename the identity at the same time.
Use DeleteBeforeReplace: Ensure that the old role assignment is deleted before the new one is created.

Here is how you can implement this in Pulumi using C#:

using Pulumi;
using Pulumi.AzureNative.Authorization;
using Pulumi.AzureNative.Authorization.Inputs;
using Pulumi.AzureNative.ManagedIdentity;

class MyStack : Stack
{
    public MyStack()
    {
        var identity = new UserAssignedIdentity("newIdentity", new UserAssignedIdentityArgs
        {
            ResourceGroupName = "my-resource-group",
            // Other identity properties
        });

        var roleAssignment = new RoleAssignment("newRoleAssignment", new RoleAssignmentArgs
        {
            PrincipalId = identity.PrincipalId,
            RoleDefinitionId = "/subscriptions/{subscriptionId}/providers/Microsoft.Authorization/roleDefinitions/{roleDefinitionId}",
            Scope = "/subscriptions/{subscriptionId}/resourceGroups/my-resource-group",
        }, new CustomResourceOptions
        {
            DeleteBeforeReplace = true
        });
    }
}

In this example, DeleteBeforeReplace is explicitly set to true, ensuring that Pulumi deletes the old role assignment before creating the new one. Note that DeleteBeforeReplace needs to already be in Pulumi state for it to take effect. So a rename takes place in the same commit/revision when enabling DeleteBeforeReplace, it won’t take effect.

2. Renaming Identity Without Renaming Role Assignments

If you only rename the identity but leave the role assignments unchanged, you can run into issues such as:

Updating SQL Role Assignment Principal ID is not permitted: This error occurs because Azure does not allow direct updates to the PrincipalId of a role assignment.

To avoid this, you must also rename the role assignment when renaming the identity:

using Pulumi;
using Pulumi.AzureNative.Authorization;
using Pulumi.AzureNative.Authorization.Inputs;
using Pulumi.AzureNative.ManagedIdentity;

class MyStack : Stack
{
    public MyStack()
    {
        var identity = new UserAssignedIdentity("updatedIdentity", new UserAssignedIdentityArgs
        {
            ResourceGroupName = "my-resource-group",
            // Other identity properties
        });

        // Role assignment must be renamed alongside the identity
        var roleAssignment = new DocumentDB.SqlResourceSqlRoleAssignment("updatedRoleAssignment", new DocumentDB.SqlResourceSqlRoleAssignmentArgs
        {
            PrincipalId = identity.PrincipalId,
            RoleDefinitionId = "/subscriptions/{subscriptionId}/providers/Microsoft.Authorization/roleDefinitions/{roleDefinitionId}",
            Scope = "/subscriptions/{subscriptionId}/resourceGroups/my-resource-group",
        }, new CustomResourceOptions
        {
            DeleteBeforeReplace = true
        });
    }
}

Here, the role assignment is renamed to match the identity’s new name, and DeleteBeforeReplace ensures that the transition happens smoothly without encountering errors.

Potential Pitfalls

State File Considerations: If DeleteBeforeReplace is already in the state, you might not encounter the “role already exists” error because the resource will be replaced cleanly. However, always double-check your resource states and configurations, especially when working in a collaborative environment where the state might have been modified by others.
Resource Dependencies: Ensure that your Pulumi code correctly models dependencies between resources. For instance, role assignments should depend on the existence of the identity, which Pulumi typically handles, but it’s good practice to verify this.

Conclusion

Handling renaming of identities and role assignments in Pulumi requires careful planning to avoid common pitfalls. By using DeleteBeforeReplace and ensuring that associated resources like role assignments are renamed together, you can avoid errors and ensure a smooth deployment process. Pulumi’s flexibility with C# allows you to manage these intricacies effectively, enabling reliable infrastructure provisioning in Azure.

Leveraging ephemeral image registry ttl.sh using Azure Pipelines

Wed, 10 Apr 2024 17:00:00 GMT

Introduction

When it comes to testing Docker images, having a reliable platform to quickly spin up containers for testing purposes can be invaluable. Enter ttl.sh, a convenient service that allows you to host and test Docker images with ease. In this blog post, we’ll explore how to leverage ttl.sh within Azure Pipelines to streamline your testing workflow.

What is ttl.sh?

Ttl.sh is a service designed to simplify Docker image testing. It provides a temporary container registry (hence the name “ttl,” which stands for “time to live”) where you can push your Docker images for testing purposes. Once pushed, ttl.sh automatically spins up containers from your images, allowing you to validate their functionality and compatibility. Bear in mind that the image will get purged after 24hrs or less depending on the tag.

Setting Up Azure Pipelines with ttl.sh

To integrate ttl.sh into your Azure Pipelines workflow, follow these steps:

Create Dummy Credentials in Azure DevOps Service Connection: Start by creating a dummy service connection in Azure DevOps. This connection will be used to authenticate and push Docker images to ttl.sh. While the credentials don’t need to be real, ensure they have the necessary permissions to push images to your chosen container registry.

Configure Your Pipeline: Below is a basic example of an Azure Pipelines YAML configuration that builds a Docker image and pushes it to ttl.sh:

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- task: Docker@2
  inputs:
    containerRegistry: 'ttlsh'
    repository: '2cc2bc2c-cc5d-4c47-81ed-6ea9aaead977'
    command: 'buildAndPush'
    Dockerfile: './Dockerfile'
    tags: '8h'

In this example:

The pipeline triggers on changes to the main branch.
It utilizes an Ubuntu latest image.
The Docker task builds the image using the specified Dockerfile and pushes it to the ttl.sh registry. The tags parameter specifies how long the image should be available on ttl.sh before expiring.
Customize as Needed: Customize the pipeline according to your project’s requirements. You can add additional steps for testing, deployment, or any other tasks necessary for your workflow.

Conclusion

Integrating ttl.sh into your Azure Pipelines workflow provides a convenient solution for testing Docker images. By following the steps outlined above, you can easily push Docker images to ttl.sh and validate their functionality within temporary containers. This streamlined testing process helps ensure the reliability and compatibility of your Docker images before deploying them to production environments. Happy testing!

Simplifying Resource Management With Parent Option in Pulumi

Thu, 28 Mar 2024 23:30:00 GMT

Introduction

In the world of infrastructure as code (IaC), managing dependencies and relationships between resources is crucial for maintaining an efficient and organized infrastructure. Pulumi, a powerful IaC tool, offers a feature called the “Parent Option” to streamline resource management within its ecosystem. In this post, we’ll explore the Parent Option in Pulumi and understand how it simplifies resource orchestration through a practical example.

What is the Parent Option?

The Parent Option in Pulumi allows developers to specify the parent-child relationship between resources. When creating resources within Pulumi, specifying a parent resource helps in organizing and managing dependencies effectively. By defining parent-child relationships, Pulumi can automate resource lifecycle management, ensuring proper ordering of resource creation, updates, and deletion.

Practical Example: Custom Storage Account

Let’s consider a scenario where we need to create a custom storage account in Azure using Pulumi. We’ll create a custom component resource called CustomStorageAccount that encapsulates the creation of a storage account. Initially, we’ll create the resource without specifying a parent, and then we’ll refactor the code to utilize the Parent Option.

public class CustomStorageAccount : ComponentResource
{
    public CustomStorageAccount(string name,
        CustomStorageAccountArgs args,
        ComponentResourceOptions? options = null)
        : base("CommonComponent:CustomStorageAccount", name, options)
    {
        // Creating storage account without specifying a parent
        var storageAccount = new StorageAccount("sa", new StorageAccountArgs
        {
            ResourceGroupName = args.ResourceGroupName,
            Sku = new SkuArgs
            {
                Name = SkuName.Standard_LRS
            },
            Kind = Kind.StorageV2,
            Tags = new InputMap<string> {
                { "Purpose", args.Purpose },
                { "Owner", args.Owner }
            }
        });
    }
}

When previewing this configuration, resources will be created independently, without any parent-child relationship:

pulumi preview --stack stacknamedev01
Previewing update (stacknamedev01):
     Type                                     Name                            Plan
 +   pulumi:pulumi:Stack                      stackname-stacknamedev01  create
 +   ├─ CommonComponent:CustomStorageAccount  sa                              create
 +   ├─ azure-native:resources:ResourceGroup  resourceGroup                   create
 +   └─ azure-native:storage:StorageAccount   sa                              create
Resources:
    + 4 to create

Utilizing the Parent Option

Now, let’s refactor the code to utilize the Parent Option by specifying the component resource (CustomStorageAccount) as the parent for the storage account resource. Note the new CustomResourceOptions { Parent = this }.

public class CustomStorageAccount : ComponentResource
{
    public CustomStorageAccount(string name,
        CustomStorageAccountArgs args,
        ComponentResourceOptions? options = null)
        : base("CommonComponent:CustomStorageAccount", name, options)
    {
        // Creating storage account with parent specified
        var storageAccount = new StorageAccount("sa", new StorageAccountArgs
        {
            ResourceGroupName = args.ResourceGroupName,
            Sku = new SkuArgs
            {
                Name = SkuName.Standard_LRS
            },
            Kind = Kind.StorageV2,
            Tags = new InputMap<string> {
                { "Purpose", args.Purpose },
                { "Owner", args.Owner }
            }
        }, new CustomResourceOptions { Parent = this });
    }
}

When previewing this configuration, the parent-child relationship is correctly inferred, and resources will be organized accordingly:

pulumi preview --stack stacknamedev01
Previewing update (stacknamedev01):
     Type                                       Name                            Plan
 +   pulumi:pulumi:Stack                        stackname-stacknamedev01  create
 +   ├─ CommonComponent:CustomStorageAccount    sa                              create
 +   │  └─ azure-native:storage:StorageAccount  sa                              create
 +   └─ azure-native:resources:ResourceGroup    resourceGroup                   create
Resources:
    + 4 to create

Conclusion

The Parent Option in Pulumi simplifies resource management by allowing developers to define parent-child relationships between resources. This feature enhances the organization and maintainability of infrastructure code, leading to more robust and scalable infrastructure deployments.

Resolving NU1301 when using Azure Artifacts

Wed, 14 Feb 2024 14:30:00 GMT

Understanding the NU1301 Error

Before we delve into the solution, let’s briefly understand what the NU1301 error signifies. This error message typically occurs when the NuGet package manager is unable to load the service index for a particular package source. It can happen due to various reasons, such as network connectivity issues, misconfigured package sources, or authentication problems. In this post we will look to solve the authentication problem.

Step 1: Install the Azure Artifacts Credential Provider

The Azure Artifacts Credential Provider is a tool that helps manage credentials for Azure Artifact based NuGet package sources. To install this provider, follow docs.

Step 2: Run `dotnet restore --interactive`

Once installed, run the dotnet restore --interactive command to resolve the NU1301 error. This command initiates an interactive authentication process, allowing us to provide credentials for the NuGet package sources that require authentication. Follow these steps:

Navigate to your .NET project directory using the command prompt or terminal.
Run the following command:
```
dotnet restore --interactive
```
When prompted, enter the required credentials for the NuGet package sources. Make sure to provide accurate credentials to authenticate successfully.
Once the authentication process is complete, the dotnet restore command will proceed to restore the NuGet packages for your project without encountering the NU1301 error.

Conclusion

This post is more like a note to self and I hope it helps out if others get this error. Feel free to reach out if you have any questions or encounter any issues along the way. Happy coding.

Unity Android - Resolve APK Failed to Install

Sun, 11 Feb 2024 22:30:00 GMT

As a newcomer to the world of Android development and Unity, encountering errors can be a daunting experience. One common stumbling block that many beginners face is the dreaded “APK failed to install” error, accompanied by the message: “INSTALL_FAILED_NO_MATCHING_ABIS: Failed to extract native libraries, res=-113.” If you’ve encountered this error, fear not! In this guide, we’ll walk through a simple solution that involves adjusting some settings in Unity to resolve the issue.

Before we dive into the solution, let’s first understand what this error means. When we build our Unity project for Android, it includes native libraries specific to the CPU architecture of the target device. These native libraries are compiled into our APK and are essential for our app to run correctly on different devices. The error message “INSTALL_FAILED_NO_MATCHING_ABIS” indicates that the Android Package (APK) does not contain native libraries compatible with the CPU architecture of the device on which we are trying to install the app.

To fix this error, follow these steps:

Open our Unity Project: Launch Unity and open the project in which we are encountering the “APK failed to install” error.
Navigate to Player Settings: In the Unity Editor, go to Edit > Project Settings > Player to access the Player Settings for our project.
Set Scripting Backend to IL2CPP: In the Player Settings window, locate the “Other Settings” section. Under this section, find the “Scripting Backend” dropdown menu and select “IL2CPP.” IL2CPP is a Unity technology that translates our C# scripts into C++ code, which can then be compiled natively for the target platform, including Android.
Select ARMv7 and ARM64 Architectures: Still in the Player Settings window, scroll down to the “Target Architectures” section. Check the boxes next to “ARMv7” and “ARM64” architectures. These options ensure that our APK includes native libraries compatible with both ARMv7 and ARM64 CPU architectures, covering a wide range of Android devices.
Rebuild our Project: After making these changes, rebuild our Unity project for Android by going to File > Build Settings > Build. Ensure that we select the appropriate platform (Android) and click on “Build.” This process will generate a new APK with the updated settings.
Test our App: Once the build process is complete, install the newly generated APK on our device and verify that the “APK failed to install” error no longer occurs. our app should now install and run successfully on a variety of Android devices.

By setting the scripting backend to IL2CPP and selecting ARMv7 and ARM64 architectures in the Unity Player Settings, we’ve ensured that our APK includes the necessary native libraries to support a wide range of Android devices. This simple adjustment resolves the “INSTALL_FAILED_NO_MATCHING_ABIS” error and paves the way for a smoother development experience.

Remember, as a newcomer to Android and Unity development, I am learning this solution among few others. This post is mainly for me not to forget this lesson and hopefully it helps ou out other newbies as well. Happy coding!

Using Pulumi's Apply Method in C#

Thu, 11 Jan 2024 10:30:00 GMT

Introduction

Pulumi, a powerful Infrastructure as Code (IaC) tool, empowers developers to define, deploy, and manage cloud infrastructure using familiar programming languages such as C#. One of the key features that sets Pulumi apart is its Apply method, which allows for dynamic configuration and transformation of resources during deployment. In this blog post, we’ll explore the Apply method and provide examples of its usage in C# with Pulumi.

Understanding the Apply Method

The Apply method in Pulumi serves as a bridge between resource creation and configuration. It enables developers to apply transformations and logic to the attributes of a resource before or after its creation. This is particularly useful when you need to dynamically set properties based on other resources or external inputs.

Syntax:

TResource.Apply<TResourceType>(Func<TResourceType, TResourceType> transformation)

Here, TResource represents the type of the resource, and TResourceType represents the type of the resource’s properties.

Examples of Apply Method Usage

Here are a couple of examples demonstrating the usage of the apply method in C# with Pulumi:

Concatenating Strings:

using Pulumi;

class MyStack : Stack
{
    public MyStack()
    {
        var prefix = "Hello, ";
        var name = "Pulumi";

        var greeting = Output.Create($"{prefix}").Apply(p => p + name);

        var myResource = new SomeResource("exampleResource", new SomeResourceArgs
        {
            Message = greeting,
            // other properties...
        });
    }
}

In this example, the apply method is used to concatenate the prefix and name strings to create a dynamic greeting.

Conditionally Setting Values:

using Pulumi;

class MyStack : Stack
{
    public MyStack()
    {
        var isProduction = true;

        var environment = Output.Create("development").Apply(p => isProduction ? "production" : p);

        var myResource = new SomeResource("exampleResource", new SomeResourceArgs
        {
            Environment = environment,
            // other properties...
        });
    }
}

Here, apply is employed to conditionally set the environment variable based on the value of isProduction. This allows for flexible resource configuration depending on the deployment context.

These examples showcase how the apply method in Pulumi enables dynamic and expressive infrastructure configurations in C#.

These examples demonstrate how the Apply method can be applied to Azure cloud operations and regular string operations, providing a flexible and dynamic approach to infrastructure as code in Pulumi.

Conclusion

The Apply method in Pulumi is a powerful tool for dynamic resource configuration and transformation. By leveraging this method, developers can write more flexible and dynamic infrastructure code, making it easier to adapt to changing requirements and environments. These examples showcase just a glimpse of the capabilities that Pulumi’s Apply method brings to the table, offering a seamless way to customize your infrastructure deployments in C#. For more details visit Pulumi Docs

Azure Pipelines Artifact Download Issue - A Quick Fix

Mon, 08 Jan 2024 10:30:00 GMT

The Problem:

There is a peculiar behavior when downloading artifacts using Azure Pipelines. The artifact, when viewed on the repository, displayed a seemingly correct structure. However, upon downloading, an unexpected string was prefixed to every element, causing a disruption in the intended folder structure and file names.

For a detailed look into the issue, check out the GitHub thread here.

Understanding the Solution:

My co-worker Dan Ponting found the issue as well as shared this solution with me, the details are below.

The standard way of calling the task, as shown below, led to the unwanted manipulation of folder structures and file names:

- task: DownloadPipelineArtifact@2
  inputs:
    artifactName: 'MyFavouriteArtifact'

The Solution:

The workaround involves using the shorthand version of the task, which remarkably eliminates the issue. Instead of the conventional method, the user suggests opting for the following configuration:

- download: current
  artifact: MyFavouriteArtifact

This simple adjustment in the way the task is invoked resolves the problem, ensuring that the downloaded artifact retains the intended structure without any unexpected string prefixes.

Conclusion:

The solution was easier to locate all thanks to GitHub and my coworker. The shared knowledge and experiences within the community contribute significantly to the overall improvement and efficiency of DevOps processes. Hopefully this post saves you some time as well.

For more information on the Azure Pipelines task and its configurations, refer to the official documentation here.

Making an Azure Pipeline Stage Non-Cancellable

Thu, 28 Dec 2023 10:30:00 GMT

Introduction

In a recent StackOverflow post, I sought advice on making a specific stage in an Azure Pipeline non-cancellable. This blog explores the question and summarizes the solutions proposed on the post as well as my co-worker Dan.

The Challenge

Developers often face the challenge of ensuring the reliability of each stage in an Azure Pipeline. My specific concern was making a stage non-cancellable, meaning it should not be interrupted once started.

Understanding Azure Pipelines

Azure Pipelines involve stages, each comprising one or more jobs. Jobs represent phases with multiple tasks. My focus was on enforcing a non-cancellable behavior for a particular stage.

Proposed Solution

Custom Conditions and PowerShell Scripts

Use always() to control stage behavior. A condition that always evaluates to true ensures a stage runs, making it non-cancellable.

stages:
- stage: CustomConditionStage
  jobs:
  - job: CustomConditionJob
    condition: always()
    steps:
    - powershell: |
        Write-Host "Running CustomConditionStage"

Conclusion

Making an Azure Pipeline stage non-cancellable is important in cases where we share use shared resources. Using Pulumi or Terraform state file backend storage for example or running tests on real cloud resources. In such cases we need to ensure such stages complete successfully and finish their cleanup before next one starts. In such cases the always() function and exclusive locks are really handy.

Pass app configuration via docker entrypoint

Sun, 09 Jul 2023 00:00:00 GMT

Introduction

There is a nice post on the many available ways of passing in application configuration into a docker container. Here is another one, which is similar to the ones listed in the post but still possibly stands out as a different technique.

App config via entrypoint

The idea is that we pass in environment variables to a script that runs as the container entrypoint. This entrypoint script generates a settings JSON file for the application to use as its configuration.

Docker file with nginx base (can be any other base, but this example is nginx specific)

FROM nginx:1.23.4-alpine-slim
COPY ./ /src/app/
WORKDIR /src/app
RUN chmod +x settings.sh
ENTRYPOINT ["sh","/src/app/settings.sh"]

The settings.sh

#!/bin/sh
# expecting these variables are set via environment
echo "{ \"SETTING1\": \"$SETTING1\", \"SETTING2\": \"$SETTING2\" }" > settings.json

# display file to screen for verification
cat settings.json

echo "running nginx command"
set -e
exec nginx -g "daemon off;"
echo "finished running nginx command"

The exec nginx -g "daemon off;" is the crucial bit. As we are providing an entrypoint, the entrypoint of the base image won’t run. So we need to do what we expect out of the base.

Docker build, say we call this image dockerpoc

docker build -t dockerpoc:1.0 --no-cache .

Finally, we pass in the required values via environment variables during the docker run.

docker run -e SETTING1='setting1-value' -e SETTING2='setting2-value' -it dockerpoc:1.0

This creates the configuration for the application in a JSON format at the docker container runtime.

Conclusion

Hope this was useful and saves you some time. Please do share your learnings. If you have any thoughts or comments, please do get in touch with me on Twitter @rubberduckdev. Or use the Disqus plugin below.

Credits

Banner by Rubaitul Azad

Unix line endings on Windows

Sat, 08 Jul 2023 23:30:00 GMT

Introduction

Many of us are likely running Linux-based container images on the Windows operating system. This is where writing scripts in a Windows IDE such as VSCode and then running them in a container image such as Alpine or Debian becomes troublesome due to encoding and line endings. I recently hit this line-ending issue and took a while to understand the following. There are two types of line endings based on the two main operating systems. Linux and Mac OS X systems use the Unix line ending, which is LF. Windows systems use the DOS line ending, which is CR+LF. When we have the Windows-style line ending and try to run the script in a Linux-based docker container, it ends up throwing errors.

Workaround/Solution

The workaround I used was to install WSL Ubuntu. This is so that I can test run any shell scripts I write before running them in a docker container.

wsl --install Ubuntu-22.04

Install a tool called dos2unix, and it has a detailed man page as well.

rubberduck@MYUNIXMACHINE:/mnt/c$sudo apt-get update
rubberduck@MYUNIXMACHINE:/mnt/c$sudo apt-get install dos2unix

Finally, use the dos2unix tool to change the encodings of the script (say script.sh as an example) so it can run on Linux.

rubberduck@MYUNIXMACHINE:/mnt/c$ cd source
rubberduck@MYUNIXMACHINE:/mnt/c$ dos2unix script.sh

Conclusion

There might be other better ways of achieving this, and I am still learning the Unix ways of life. If you are aware of an easier technique, please share. Hope this was useful and saves you some time. Please do share your learnings. If you have any thoughts or comments, please do get in touch with me on Twitter @rubberduckdev. Or use the Disqus plugin below.

Credits

Banner by Gabriel Heinzer

Azure pipelines - extending yaml array

Mon, 29 May 2023 15:30:00 GMT

Introduction

Azure Pipelines is a continuous integration and continuous delivery (CI/CD) service that helps you automate the building, testing, and deploying of your code to any target environment. I like using the yaml pipelines so that we are practising true GitOps.

Overall scenario

In a bid to reduce repetition, it is a good idea to create and share Azure pipeline yaml templates. Let’s say we have a template as follows:

parameters:
    - name: dockerPushDependency
      type: object
      default: []  
  stages:
    - stage: DockerBuild
      pool: AgentPoolWithDockerInstance
      jobs:
        - job: DockerBuild
          steps:
          - task: Docker@2
            name: DockerBuild
            displayName: Build image to container registry
            inputs:
              repository: MyRepository
              command: Build
              containerRegistry: MyContainerRegistry
              tags: tag1

    - stage: DockerImageScan
      # ignore details, the built image gets scanned here for known vulnerabilities

    - stage: DockerPush
      dependsOn:
      # needs to depend on parameters and the image scan stage

      pool: AgentPoolWithDockerInstance
      jobs:
        - job: DockerPush
          steps:
          - task: Docker@2
            name: DockerPush
            displayName: Push the image to the container registry
            inputs:
              repository: MyRepository
              command: Push
              containerRegistry: MyContainerRegistry
              tags: tag1

In our template, we have 3 stages. We pass in dependencies into the template so that the docker push can check a previous step, say a test stage, has been completed before it can trigger. Complexity arises regarding the dependency of DockerPush as it needs to depend on an array of stages that have been passed in as well as a known DockerImageScan stage which is within the template. How do we merge or add this value into the provided array in Azure pipelines yaml?

The solution

The solution that worked is based on a great idea provided on Stackoverflow.

- stage: DockerPush
      dependsOn:
      - ${{ each dependencyItem in parameters.dockerPushDependency }}:
        - ${{ dependencyItem }}
        - DockerImageScan

Instead of performing any array manipulation, we just loop through it and present the values to the DockerPush dependsOn parameter. This stage can now depend on whatever the consumer has provided as well as the known stage within the pipeline template.

Conclusion

Although the trick is quite simple, took hours to get to. Mainly the feedback cycle is slow with Azure pipelines. There is hope though, given such a feature is in preview now. Overall, hope this post helps you out with Azure pipelines and happy CI CD to you. Please do share your learnings. If you have any thoughts or comments please do get in touch with me on Twitter @rubberduckdev. Or use the Disqus plugin below.

Azure container app - Kestrel - Http 431

Sat, 27 May 2023 23:30:00 GMT

Introduction

This post is about an error over which I spent hours trying to get anywhere until my colleague Sebastian Wiejas spotted the issue and suggested the solution.

Overall scenario

We have a dotnet api deployed as a docker image which is hosted on Azure Container Apps instance. But whenever we would try to hit an endpoint, from the UI app (which is hosted as a separate) app, we will get the following error

Request URL: https://testapi.containerapp.io/api/v1/endpoint1
Request Method: OPTIONS
Status Code: 431 
Remote Address: 25.86.295.132:443
Referrer Policy: strict-origin-when-cross-origin

Further investigating into Http 431 status code, it says 431 Request Header Fields Too Large, which implies we are sending too many headers or something is limiting our allowed header count.

The solution or maybe the workaround

Given it is a dotnet app and we know from the deployment image that it is using kestrel web server, so we investigated further into Kestrel settings.

One of the settings that stood out was KestrelServerLimits.MaxRequestHeaderCount Property. Although it defaults to 100, in our case, as it turns out it was set to a low number (I think 10) in the app’s appsettings.json. So the way we got the app to work is to allow more headers by setting the MaxRequestHeaderCount property in appsettings.json

{
  "Kestrel": {
    "Endpoints": {
      "Https": {
        "Url": "https://*:443",
        "Certificate": {
        }
      },
      "Http": {
        "Url": "http://*:80"
      }
    },
    "Limits": {
      "MaxRequestHeaderCount": 50 // Default is 100
    }
  }
}

Conclusion

Hope this was useful and saved you some time. And again many thanks to Sebastian for sharing these details. Overall, the app being deployed as a container on Azure container app doesn’t have anything to do with the issue. It was more the server settings which broke cors requirements. Please do share your learnings. If you have any thoughts or comments please do get in touch with me on Twitter @rubberduckdev. Or use the Disqus plugin below.

Dependabot GitHub workflow fails - deployment_token was not provided

Mon, 16 Jan 2023 15:30:00 GMT

Introduction

The automated update tool Dependabot is a very helpful bot and it creates many fixes in the form of pull requests. Recently I learnt why the dependabot pull requests were failing with an error

deployment_token was not provided.

The setup and issue

The github workflow is deploying an Azure static webapp.

- name: Deploy
        id: deploy
        uses: Azure/static-web-apps-deploy@v1
        with:
          azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_API_TOKEN_DELIGHTFUL_GLACIER_02ECCB203 }}
          repo_token: ${{ secrets.GITHUB_TOKEN }} # Used for Github integrations (i.e. PR comments)
          action: "upload"
          ###### Repository/Build Configurations - These values can be configured to match your app requirements. ######
          # For more information regarding Static Web App workflow configurations, please visit: https://aka.ms/swaworkflowconfig
          app_location: "/public" # App source code path
          skip_app_build: true
          skip_api_build: true
          ###### End of Repository/Build Configurations ######

This means that it needs a valid access token to be able to create resources on Azure during the workflow execution. And it is unable to find that secret and hence the error:

deployment_token was not provided.
The deployment_token is required for deploying content. If you'd like to continue the run without deployment, add the configuration skip_deploy_on_missing_secrets set to true in your workflow file
An unknown exception has occurred

The solution

Took me a few search attempts to locate this comment on GitHub. Essentially, dependabot doesn’t use the default set of secrets, rather we need to explicitly provide a set of secrets for it to use. It makes sense, we don’t want an app sitting outside of our repo to have access to repo/organization secrets.

The solution is to provide the required secrets for dependabot to use:

Conclusion

Hope this was useful and saved you some time. Please do share your learnings. If you have any thoughts or comments please do get in touch with me on Twitter @rubberduckdev. Or use the Disqus plugin below.

Why is Azure firewall allowing a request that should be denied?

Sat, 14 Jan 2023 16:00:00 GMT

Introduction

Azure Firewall is an exciting cloud native network security service and provides a lot of features to make our life easier while thwarting security threats out there. While setting up the firewall policies, one thing we are fairly confident of is that Azure Firewall denies all traffic by default. And this has been documented under Configure Azure Firewall rules.

But things get a bit weird, let’s see how.

Anomaly?

With the assumption that Azure firewall is denying all requests by default, let’s setup the following policy

{
    "name": "Allow my vm to another vm",
    "ipProtocols": [
        "TCP"
    ],
    "destinationPorts": [
        "6000"
    ],
    "sourceAddresses": [
        "10.250.26.36"
    ],
    "sourceIpGroups": [],
    "ruleType": "NetworkRule",
    "destinationIpGroups": [],
    "destinationAddresses": [
        "10.200.20.30"
    ],
    "destinationFqdns": []
}

So here we are saying a certain IP is allowed to connect to another certain IP over port 6000. Which is great.

Now if we run from the vm with IP 10.250.26.36

Test-NetConnectivity -computer 10.200.20.30 -port 6000

we expect and get a success. Great!

But then if we run

Test-NetConnectivity -computer 10.200.20.30 -port 443

we should expect a failure. This is because port 443 is not allowed. But! But! But! the command succeeds. Added to this astonishment is that no logs appear in the firewall logs which show the port 443 request was allowed.

What is going on! (◎_◎;)

Did the firewall just allow an authorized request?
What happened to the log?

The explanation

TCP ping is a unique use case where if there is no allowed rule, the Firewall itself responds to the client’s TCP ping request even though the TCP ping doesn’t reach the target IP address/FQDN. Test-NetConnection does a TCP ping. This is documented.

Regarding logging, records are logged only when a specific rule match occurs. That is, Each new connection that matches one of your configured network rules results in a log for the accepted/denied connection.. I managed to get some clarifications from MS Support and was informed that this specific match is needed in case of TCP ports 80 and 443. For other ports, the denies will get logged even if there is no specific rule match.

So overall, the firewall is behaving as expected and denying traffic. But it is not logging it as there is no specific rule match.

Conclusion

One way to get uniform results and gain more confidence will be to have a low priority blanket deny all firewall policy. So, if no allows are matched, a request gets denied and then logged in firewall logs. Although it took me a while to figure this out, was relieved that no unauthorized requests were being allowed by the firewall.

ExpressRoute missing spoke route

Sat, 14 Jan 2023 15:00:00 GMT

Introduction

After an Azure ExpressRoute is setup we can use Get-AzExpressRouteCircuitRouteTable Cmdlet to view what routes are advertised on the ExpressRoute. It lists the destinations a data packet can reach if it took that route. And those data packets take the ExpressRoute only when the destination is available on the route table.

Recently, my team and I came across this issue where a certain spoke IP range wasn’t listed on the route table.

ExpressRoute route table in a hub-spoke network topology

So while using hub-spoke model say our setup is as follow:

Hub vnet is 10.10.0.0/22
Spoke1 vnet is 10.20.0.0/22
Spoke2 vnet is 10.30.0.0/22

Say our on-premises IP ranges are as follows, to which ExpressRoute is the path from our Azure infrastructure:

162.20.0.0/21

Now traffic from a spoke, say Spoke1 will flow to on-prem and back only if the ExpressRoute route table has entries for 10.20.0.0/22and 162.20.0.0/21 respectively. These routes are learnt by the ExpressRoute automatically once we deploy the vnet peerings.

But when we run the Get-AzExpressRouteCircuitRouteTable Cmdlet we see

The issue

After setting up the infrastructure as described above, traffic was not flowing from on-premises to a resource on Spoke1. The issue was understandable as the ExpressRoute circuit table wasn’t showing entries for the spokes.

i.e.

Get-AzExpressRouteCircuitRouteTable -DevicePath Primary -ExpressRouteCircuitName my-express-route -PeeringType AzurePrivatePeering -ResourceGroupName my-rg

Network : 162.20.0.0/21
NextHop : 10.300.120.250

So in the output above, we see that ExpressRoute knows that to get to the on-prem resources, it can use a next hop (which in this case is the ExpressRoute private peering). But it doesn’t know how to get to Azure resources from on-prem. This could manifest in multiple ways while testing, such as:

Invoke-WebRequest : The underlying connection was closed: The connection was closed unexpectedly.

Invoke-WebRequest : Unable to connect to the remote server

Now we understand the reason, let’s see what is the solution.

Solution - check vnet peering

The hub to spoke vnet peering should look like this:

The issue we found is that the spokes were not using the hub's gateway. This is a setting we need on the spoke side of vnet peering. And by default, it is set to none.

Once we have got the spokes to use the hub’s gateway, the resulting route table for ExpressRoute should show a NextHop for Spoke vnet IP ranges as below. The NextHop being the hub vnet gateway IP.

Get-AzExpressRouteCircuitRouteTable -DevicePath Primary -ExpressRouteCircuitName my-express-route -PeeringType AzurePrivatePeering -ResourceGroupName my-rg

Network : 162.20.0.0/21
NextHop : 10.300.120.250

Network : 10.20.0.0/22
NextHop : 10.10.0.10

Network : 10.30.0.0/22
NextHop : 10.10.0.10

So the solution is to set Use the remote virtual network's gateway or Route Server using your favourite IaC.

Conclusion

Azure DevOps service connection with Service Principal using Certificate

Fri, 23 Sep 2022 16:30:00 GMT

Banner from wixstatic

Introduction

Azure DevOps service connection to Azure, using a Service Principal can use a secret (a password) or a certificate. In this post, I am sharing how I recently used a certificate to achieve authentication for a service connection. As described in this MS Community post, we can use a certificate (AsymmetricX509Cert) with any type of issuers like Self-Sign, Public or Internal CA. Self-signed is the easiest approach, in my opinion, it mimics the same logic of us setting a password and resetting it when it expires.

Generating a self-signed certificate

There is a difference between the certificates we upload for Azure Service Principal and the Azure DevOps service connection.

Azure Service Principal Certificate

We need to upload a certificate (public key) with one of the following file types: .cer, .pem, .crt. And it has to be the public key only.

Azure DevOps Service Connection Certificate

In Azure DevOps service connection we need to provide PEM file content. Include both certificate and private key content. This is different to what we uploaded onto the Azure service principal.

Generation script

Pre-requisite

We need OpenSSL installed. It can be installed using Chocolatey.

The script

So I followed a few blogs such as Certificate-based auth with Azure Service Principals from Linux command line and Azure DevOps – use certificate for Azure Service Connection SPN and put together the following PowerShell script which generates the certificates in the correct format and also shows a helpful message on where to use which one:

Github gist available.

param (
    [string]$CertificateNamePrefix,
    [int]$ExpiryInDays
)

$unsecurePassword = Read-Host "Enter password for certificate" -AsSecureString

$certFileName = "$CertificateNamePrefix-cert.pem"
$certPrivateKeyFileName = "$CertificateNamePrefix-key.pem"
$certPackFileName = "$CertificateNamePrefix-pack.pfx"
$certPemWithBagAttributesFileName = "$CertificateNamePrefix-PemWithBagAttributes.pem"

# generate cert.pem to be uploaded for spn
openssl req -x509 -days $ExpiryInDays -newkey rsa:2048 -keyout $certPrivateKeyFileName -out $certFileName
Read-Host “Check that $certFileName and $certPrivateKeyFileName are generated. Then press ENTER to continue...”

# generate pack to put private key and cert (public key) together
openssl pkcs12 -inkey $certPrivateKeyFileName -in $certFileName -export -out $certPackFileName -passout "pass:$unsecurePassword"
Read-Host “Check that $certPackFileName is generated. Then press ENTER to continue...”

# generate merged pem for pasting into Azure DevOps servce connection configuration
openssl pkcs12 -in $certPackFileName -passin "pass:$unsecurePassword"  -out $certPemWithBagAttributesFileName -nodes

Write-Host "Upload $certFileName to Azure Service principal."
Write-Host "Copy and paste content of $certPemWithBagAttributesFileName in Azure DevOps service connection certificate textbox."

For example, a test run can look like this:

PS C:\> .\GenerateSelfSignedCertificate.ps1 -CertificateNamePrefix TestAdos -ExpiryInDays 730
Enter password for certificate: ******
Generating a RSA private key
.............+++++
.......................................................................+++++
writing new private key to 'TestAdos-key.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:UK
State or Province Name (full name) [Some-State]:.
Locality Name (eg, city) []:.
Organization Name (eg, company) [Internet Widgits Pty Ltd]:.
Organizational Unit Name (eg, section) []:.
Common Name (e.g. server FQDN or YOUR name) []:.
Email Address []:.
Check that TestAdos-cert.pem and TestAdos-key.pem are generated. Then press ENTER to continue...:

Enter pass phrase for TestAdos-key.pem:
Check that TestAdos-pack.pfx is generated. Then press ENTER to continue...:

Upload TestAdos-cert.pem to Azure Service principal.
Copy and paste content of TestAdos-PemWithBagAttributes.pem in Azure DevOps service connection certificate textbox.
PS C:\dev\setup-dev-evn\AzureAutomation\self-signed-cert-generation>

Conclusion

This script is useful as it helps create both the different certificates which we need to use as well as provides a helpful message saying which one goes where. Hope this was useful and saved you some time. Please do share your learnings. If you have any thoughts or comments please do get in touch with me on Twitter @rubberduckdev. Or use the Disqus plugin below.

VNet peering using Azure Policy

Fri, 19 Aug 2022 10:30:00 GMT

Introduction

Azure policy is a service on Azure which helps us achieve organisation-wide resource governance by creating policies in Azure to govern every existing or future resource deployed. I was investigating if we can use this to reduce the workload on our infrastructure deployment and automate connectivity.

The overall idea is that use a policy definition to create a virtual network peering between hub and spokes.

Hub-spoke network topology

The hub-spoke model is a general network topology used, mainly if we want a hybrid setup. There will be a hub virtual network acting as the central point of connectivity to the on-premises network. Spoke virtual networks are used to isolate workloads in their virtual networks. Two virtual networks can be connected using something called virtual network peering.

So in this post, we can assume that the hub virtual network already exists. And we have multiple development environment virtual networks being deployed on demand. The hub can be RBAC restrictions regarding if those deployment users and processes have access. This is where azure policy steps in.

Azure policy

I had to refresh my azure policy memory and the following video helped a lot.

Policy definition

After a few trials and error, this is the policy definition that finally worked

{
    "mode": "Indexed",
    "policyRule": {
        "if": {
            "field": "type",
            "equals": "Microsoft.Network/virtualNetworks"
        },
        "then": {
            "effect": "deployIfNotExists",
            "details": {
                "type": "Microsoft.Network/virtualNetworks/virtualNetworkPeerings",
                "resourceGroupName": "testrg",
                "roleDefinitionIds": [
                    "/providers/Microsoft.Authorization/roleDefinitions/4d97b98b-1d4f-4787-a291-c67834d212e7"
                ],
                "deployment": {
                    "properties": {
                        "mode": "incremental",
                        "template": {
                            "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json",
                            "contentVersion": "1.0.0.0",
                            "parameters": {
                                "newvnetId": {
                                    "type": "string"
                                },
                                "newvnetName": {
                                    "type": "string"
                                },
                                "hubvnetId": {
                                    "type": "string"
                                },
                                "hubvnetName": {
                                    "type": "string"
                                }
                            },
                            "resources": [
                                {
                                    "type": "Microsoft.Network/virtualNetworks/virtualNetworkPeerings",
                                    "apiVersion": "2020-11-01",
                                    "name": "[concat(parameters('hubvnetName'),'/', concat(parameters('hubvnetName'),'-', parameters('newvnetName')))]",
                                    "properties": {
                                        "peeringState": "Connected",
                                        "remoteVirtualNetwork": {
                                            "id": "[parameters('newvnetId')]"
                                        },
                                        "allowVirtualNetworkAccess": true,
                                        "allowForwardedTraffic": true,
                                        "allowGatewayTransit": false,
                                        "useRemoteGateways": false
                                    }
                                },
                                {
                                    "type": "Microsoft.Network/virtualNetworks/virtualNetworkPeerings",
                                    "apiVersion": "2020-11-01",
                                    "name": "[concat(parameters('newvnetName'),'/', concat(parameters('newvnetName'),'-hub'))]",
                                    "properties": {
                                        "peeringState": "Connected",
                                        "remoteVirtualNetwork": {
                                            "id": "[parameters('hubvnetId')]"
                                        },
                                        "allowVirtualNetworkAccess": true,
                                        "allowForwardedTraffic": true,
                                        "allowGatewayTransit": false,
                                        "useRemoteGateways": false
                                    }
                                }
                            ]
                        },
                        "parameters": {
                            "newvnetId": {
                                "value": "[field('id')]"
                            },
                            "newvnetName": {
                                "value": "[field('name')]"
                            },
                            "hubvnetName": {
                                "value": "hub-vnet"
                            },
                            "hubvnetId": {
                                "value": "/subscriptions/bf97e223-050a-44b3-b142-c8f916b707f2/resourceGroups/testrg/providers/Microsoft.Network/virtualNetworks/hub-vnet"
                            }
                        }
                    }
                }
            }
        }
    }
}

The definition above creates a virtual network peering with the hub virtual network for every new virtual network deployed. The parameters section (which I am sure can be optimized) automatically gets the name and id of the new virtual network being deployed and the hub details are hard coded.

Testing Tips

Patience

Azure policies take time to get applied. It is difficult to predict when it has got applied, so I normally had to wait 10-15mins before checking if it has worked. Officially Azure portal will say it can take up to 30mins and yes it can.

ARM

Test the deployment section by deploying outside of the policy definition. Some of the errors during deployment can take 5-10mins or longer to show up.

Activity log

Check the activity log of the resource group to spot deployIfNotExists and if it has succeeded. It normally should give a useful error message if failed.

Conclusion

Finding the outbound IP address of an azure function/webapp

Mon, 01 Aug 2022 10:30:00 GMT

Introduction

It is quite useful to know the outbound IP address of an Azure function app or an app service. Both these services are the same under the hood, hence the following techniques apply to both.

Finding the outbound IP address

Check under properties for all possible ones

Under the app (function app or app service) properties we can see the list of outbound IP address that Azure can possibly use.

This tab shows a whole list of IP addresses and not the specific IP actually used for an outbound request. Also if app is vnet integrated and the subnet has a route table, these IPs won’t be used. So the best source of truth will be doing an outbound request from the app’s console. Let’s see how next.

Use console to find IP for specific request

This trick uses as free service called https://www.ipify.org/.

We go into the app’s console and type the following command:

C:\home\site\wwwroot>curl -s https://api.ipify.org

That should return the outbound IP from which the request went to ipify service.

Mimic ipipfy using own service

We can also create our own service e.g. an Azure function.

The dotnet code below can return the caller’s IP

#r "Newtonsoft.Json"

using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Newtonsoft.Json;

public static async Task<IActionResult> Run(HttpRequest req, ILogger log)
{
    string responseMessage = $"Caller ip = {req.HttpContext.Connection.RemoteIpAddress.ToString()}";
    return new OkObjectResult(responseMessage);
}

We can call the service using postman to test or from console of another app

Conclusion

Terraform - working around 'error installing provider'

Tue, 19 Jul 2022 13:30:00 GMT

Introduction

Recently many teams I have been working with have been hitting this error installing provider issue with Terraform. We have noticed that it is only happening with Terraform 0.11 (could be happening with other versions, we just haven’t noticed it yet).

The error message is in lines of

Error installing provider "null": openpgp: signature made by unknown entity. 
Terraform analyses the configuration and state and automatically downloads 
plugins for the providers used. However, when attempting to download this 
plugin an unexpected error occured. 
This may be caused if for some reason Terraform is unable to reach the 
plugin repository. The repository may be unreachable if access is blocked 
by a firewall. 
If automatic installation is not possible or desirable in your environment, 
you may alternatively manually install plugins by downloading a suitable 
distribution package and placing the plugin's executable file in the 
following directory: 
    terraform.d/plugins/windows_amd64

Workaround

Warning: This is not recommended for production use.

The only way it seems to work for us is to use unverified plugins.

terraform init -verify-plugins=false

Explanation

The issue here is the openpgp plugin signature has changed and by default Terraform performs verification for any plugins used. By skipping verification, we are at risk of getting security vulnerabilities into our deployment process. Hence it is not recommended to skip plugin verification in production systems. This can be a quick workaround while you possibly upgrade your Terraform and provider versions to use the latest valid plugins.

Please note that I was unable to find any official documentation about this terraform init parameter. The only place it is described is at lightnetics.com.

Conclusion

Using Azure App Service managed certificate

Fri, 28 Jan 2022 16:30:00 GMT

Introduction

Took me a while on how to enforce https on the websites I have deployed using Azure App Service. The main blocker for me was getting a SSL/TLS certificate. Given I have mostly hobby websites, one of the options was to get a free one from https://letsencrypt.org. But it quickly became clear from their documentation that there is a learning curve! Then I found the easy way out, Azure App Service free managed certificate.

Azure App Service managed certificate

The Microsoft documentation itself is already very good, I will try to add the quick steps I did. Once we got a custom domain validated, then on it is quite simple.

Overall there are two main steps, create the certificate and then bind your custom domain.

Create App Service Managed Certificate

Add binding to secure custom domain

Conclusion

Although it has been nicely documented, took me a while to figure that these are the two main steps needed. Hence this post, hope it helps someone else get there a bit quicker. If you have any thoughts or comments please do get in touch with me on Twitter @rubberduckdev. Or use the Disqus plugin below.

Features of an ideal IaC language

Mon, 20 Dec 2021 08:30:00 GMT

Introduction

Infrastructure as Code (IaC) is the management of infrastructure (networks, virtual machines, load balancers, and connection topology) in a descriptive model, using source code. These could be cloud infrastructure of on-prem as well. Once codified we can use all the benefits of version control, testing etc.

Festive Tech Calendar 2021

This demo is designed as my contribution for Festive Tech Calendar 2021, definitely check out the website for many other great topics. Happy Christmas.

This video is part of the festive tech calendar 2020.

Features of an ideal IaC language - my thoughts

Clarity, simplicity, unity

The language we use to deploy resources must be simple, easy to understand has a specific convention. So once written and shared, multiple users can comprehend it in the same way.

Orthogonality

Orthogonality is the property that means “Changing A does not change B”. This is nicely explained with examples in this comment on stackoverflow.

Testability

Once code is written, infrastructure or otherwise, it will add a lot to our confidence if we can test it. This makes it easier to reuse code with minimal concerns. Along with unit tests, static analysis, security testing are important. Mainly if we use IaC at scale and to help deploy real-world secured and compliant resources.

Modularity

Ideally, we won’t need to rewrite the same thing again and again. So we should be able to write once and reuse it. Hence modules in IaC are quite important.

Private registry

Registry as is, to share the modules we discuss in the previous point. A public registry is very nice to have, but a private one is a must-have in my opinion. This will help use IaC at scale, say if you have multiple teams developing and sharing modules in your organization.

Custom functions

The ability to write custom functions such as string manipulation, date manipulation etc helps configure resources easily. An ideal IaC should have support for custom functions so we won’t have to add scripts between resource deployments.

Declarative

With an IaC we should be able to say what we want and not how. For example, on Azure, we should say deploy a function app, not call a specific azure management endpoint with a specific request body.

IDE support

Good IDE experience is very important to be able to write IaC quickly and effectively.

Documentation

Although IaC has to be clear, simple and straightforward, there will be some quirks, exceptions and sometimes bugs. These features, quirks and exceptions etc of an IaC need to be documented to aid the users.

Technical support

Once an issue happens, users need to feel supported. If something with the IaC doesn’t behave as expected or is difficult to comprehend, technical support becomes very important. Nothing worse than feeling lost and not being able to get help.

Cost

The final point is cost. This is something teams will need to justify to business. If the IaC is going to hurt business, it will become difficult to adopt.

Conclusion

If you have any thoughts or comments please do get in touch with me on Twitter @rubberduckdev. Or use the Disqus plugin below.

Windows Terminal - Setup Visual Studio prompt in pwsh

Sun, 14 Nov 2021 19:30:00 GMT

Introduction

This post is more for my notes and hopefully helps someone out figure out how to set up Visual Studio environment from pwsh in Windows Terminal.

Steps

From windows terminal hit Ctrl+, to open settings pane. I normally use the json file.
Default Profile: Add pwsh profile as you default one
Then open pwsh and use command
```
code $PROFILE
```
This will open up the profile startup script
Add the following to the profile startup script.
```
$vsInstallPath = "C:\Program Files\Microsoft Visual Studio\2022\Community"
Import-Module "$vsInstallPath/Common7/Tools/Microsoft.VisualStudio.DevShell.dll"
Enter-VsDevShell -VsInstallPath $vsInstallPath -SkipAutomaticLocation -StartInPath "C:\Dev"
```
Note: The StartInPath parameter is for my local folder where I normally checkout code. Note: This is specific to VS 2022. May need changing as per VS installation.

Conclusion

Hope this was useful and saves you some time. Please do share your learnings. If you have any thoughts or comments please do get in touch with me on twitter @rubberduckdev. Or use the Disqus plugin below.

Deploying DocFX on Azure static web app

Fri, 30 Jul 2021 18:00:00 GMT

Introduction

Azure Static Web App

The Azure static web app is a modern web app service that helps with streamlined full-stack development. The feature I like most about it is the ease of CI-CD managed straight from the resource deployment. Refer documentation, Building your first static site in the Azure portal for how to.

DocFx

DocFX is a nice open-source tool to produce documentation from source code (including C#, F#, Visual Basic, REST, JavaScript, Java, Python and TypeScript) as well as raw Markdown files. It generates a static website from doc comments.

The deployment

First of all, we need a DocFx ready repo, ideally on GitHub. The simplest thing to do is to fork the DocFx seed project.

Once the repo is ready, follow the Microsoft documentation link above to create a new Azure static web app and select this repo from GitHub for Source in Deployment details.

This will create a workflow in the repo. This is super useful as it automatically handles the triggers and required steps. The only step missing will be the building of the DocFx site. This has to be added just before the autogenerated Build And Deploy step.

- name: Build Documentation
        uses: nikeee/docfx-action@v1.0.0
        with:
          args: docs/docfx.json

And that’s it.

Conclusion

This makes it very easy to have a website where you can share your API docs for other teams to refer to. And I remain very impressed with this static web app offering from Azure. You may want to refer the fork I created to test this on github.

ARM deployment - Cannot find ServerFarm

Fri, 18 Jun 2021 12:00:00 GMT

ARM Template

Deploying resources to Azure using ARM templates is generally quite easy given Azure’s native support in the generation of the template and availability of documentation. Although recently I got the following error and I spent more time than I thought I would, hence this post as a reference to myself and hopefully help others too.

The deployment error

The error I found while deploying was

{
    "Code": "NotFound",
    "Message": "Cannot find ServerFarm with name azure-functions-test-service-plan.",
    "Target": null,
    "Details": [
        {
            "Message": "Cannot find ServerFarm with name azure-functions-test-service-plan."
        },
        {
            "Code": "NotFound"
        },
        {
            "ErrorEntity": {
                "ExtendedCode": "51004",
                "MessageTemplate": "Cannot find {0} with name {1}.",
                "Parameters": [
                    "ServerFarm",
                    "azure-functions-test-service-plan"
                ],
                "Code": "NotFound",
                "Message": "Cannot find ServerFarm with name azure-functions-test-service-plan."
            }
        }
    ],
    "Innererror": null
}

The error says Cannot find ServerFarm with name azure-functions-test-service-plan. when I had the resource, also in the same resource group. Quite perplexing! After multiple attempts and then searching I found this Azure - can’t find serverfarm

Solution

The app service or function app being deployed has to be in the same region as the app service plan. And the error means it cannot locate the ServerFarm in the region of the app being deployed. So the solution is to ensure we are deploying the app to the same region as the app service plan.

I realized this after coming across this comment in the StackOverflow post linked above.

Issue was that I accidentally had the Function App in a different Azure Region than the App Service Plan it was referencing. Worth noting that this error can be caused by that.

Conclusion

This was a simple solution or possibly a user error, thanks for me not knowing that the region matters. Hope it is useful for your deployment. Thoughts and comments welcome.

Create Front Door with Rules Engine

Fri, 28 May 2021 12:00:00 GMT

Azure Front Door Rules Engine

Rules engine in an Azure front door allows us to customize how HTTP requests get handled. It is the go-to place to add CORS rules or modify caching configuration based on incoming requests and so on.

Issue with deploying Azure front door with rules engine

This is where it gets tricky. For now, I use ARM templates to deploy the front door (yet to look into Terraform/Pulumi/Bicep if they are any better at it). But this poses few issues

On the very first deployment, I found

Status: BadRequest 
Provisioning State: Failed 
Status Message: {"status":"Failed","error":{"code":"BadRequest","message":"A resource reference was invalid: \"Routing rule RoutingRule1 contains an invalid reference to RulesEngine: \"/subscriptions/abcde-ghi-479e-959c-sdhfllk/resourceGroups/rg1/providers/Microsoft.Network/frontdoors/frontdoor1/rulesengines/RulesEngine1\"\"","target":null,"details":null,"additionalInfo":null}}

Then assuming, the rules engine need to be created first, I tried adding dependson only to get:

Status: BadRequest
Provisioning State: Failed
Status Message: {"status":"Failed","error":{"code":"InvalidResource","message":"The property 'dependsOn' does not exist on type 'Microsoft.Azure.FrontDoor.Models.DeepCreatedResource_1OfFrontdoorRoutingRuleEntityV2'. Make sure to only use property names that are defined by the type.","target":null,"details":null,"additionalInfo":null}}

This is because dependson is not valid in a routine rule. So now I am out ideas.

On further investigation, turns out, it is a known bug, refer to the two below.

The suggested workaround

The suggestion on the issue is:

Right now, the following workout may work (depending on your setup for automating):

* Create Frontdoor and Rules Engine Config FIRST, without having Front door reference the Rules Engine config
* THEN, make another ARM template call, on the same Frontdoor, but added with the reference to the rules engine.

Same idea, but a different implementation of workaround

I didn’t want to have two very similar ARM templates and also maintaining ARM templates can be a pain. So as a post-deployment step, I ran some az cli:

az network front-door routing-rule update --front-door-name "frontdoor1" `
    --name "RoutingRule1" `
    --resource-group "rg1" `
    --rules-engine "RulesEngine1"

So the original ARM template just deploys the rules engine but doesn’t reference it from routing rules. The referencing happens in this post-deployment step.

Conclusion

This was a simpler solution to the issue and worked better for me. Hope it is useful for your deployment. Thoughts and comments welcome.

Simple active-active configuration with Azure Front Door

Sun, 11 Apr 2021 12:00:00 GMT

Azure Front Door

In this post, we will see a simple active-active configuration using Azure Front Door. As per Microsoft documentation, Azure Front Door is a global, scalable entry-point that uses Microsoft global edge network to create fast, secure, and widely scalable web applications. So it does a lot more than what we are looking at in this post.

Active active configuration

In the world disaster recovery, an active-active configuration is really useful as both (at least 2) services behind a load balancer are constantly in use. So if one goes down, the other one keeps serving requests.

Architecture

In our architecture diagram below, we will use two azure functions to run in an active-active mode with an Azure Front Door acting as a load balancer.

Note that both the function apps are in different regions, meaning in case of a disaster taking out one region, the other can keep working. We can span it to more regions depending on the requirements.

Function app configuration

In our case, we are deploying an HTTP triggered with anonymous authorization (to keep the setup simple). In the response message, we just specify the region it is hosted in so we can see it in our logs when we test it.

The anonymous authorization is to keep the setup simple when we test it.

Azure front door and function app configuration

On the front door, we provide a hostname/domain name which is what the users will send requests to.

In the backend pool, we add both the function apps, keeping both priority 1 and weight 50. So they will both remain active and continue servicing requests.

We do a simple pattern match /* so all requests are directed to the only backend pool we have.

Testing

We can use a small PowerShell script that keeps posting requests to our front door URL.

$requestBody = @'
{
    "name": "DP"
}
'@

do {
    $response = Invoke-WebRequest -Uri https://fdtst.azurefd.net/api/HttpTrigger1 -Method POST -Body $requestBody
    Write-Host $response.Content
} while ($true)

$response = Invoke-WebRequest -Uri https://fdtst.azurefd.net/api/HttpTrigger1 -Method POST -Body $requestBody
Write-Host $response.Content

When we first start running the script, we see it keeps getting a response from the UK south, mainly because I am closest to that data centre.

Then we go ahead and stop the UK south function app to simulate an outage from the region.

That immediately results in failing requests and this happens until the front door realizes that the backend pool is no longer available.

The azure front door keeps trying UK South, while now serving requests via UK West.

And finally, once Azure front door establishes that UK South is unhealthy, it sticks to UK West.

Conclusion

This was a simple setup to test how to implement an active-active configuration. This can be done in a far better way with an understanding of Azure SLAs to ensure we get a high availability guarantee.

Terraform - sharing modules across organization

Sat, 10 Apr 2021 12:00:00 GMT

Terraform modules

In Terraform world, modules are a way of reusing infrastructure configuration. We could create a module as a lightweight wrapper around a single resource or as a combination of multiple resources which need to be deployed regularly.

The idea

I mostly specialize in Microsoft technologies and Azure cloud, so this post will circle these technologies. The team I work in currently own multiple Terraform modules and we are working towards adopting the model shown below.

This image is from a brilliant post by Moim Hossain.

The idea is, one team, the Platform team in this case (different organizations can have a similar team named differently say, Infrastructure team, DevOps team etc.) can design modules while liaising with security, management and other stakeholders. These modules should be designed generically, tested, security analysed and then shared for consumption across product/development teams.

Apart from true IaC and GitOps, the main advantages of this model are:

Ownership: Resource deployment ownership lies with product teams. While the modules ownership lies with the Platform team.
Conventions and verifications: We have verified configurations and conventions across the organization.
Control and sanity: Platform team can maintain the overall sanity of infrastructure with controls in place such as Azure policies.
Agility: Product teams can drift away from modules under special circumstances. Or use other deployment technologies such as Pulumi, Bicep or PSArm etc.

Terraform module source

When consuming a module in Terraform we need to specify a source.

module "resource_group" {
  source = "./../modules/ResourceGroupModule"
  name     = "tfmoduletest3"
  location = "North Europe"
}

Notice the source parameter. Here it takes a local path to a terraform module. Terraform supports different modules sources. The one that we are interested in is generic Git repository as a module source.

Creating and consuming Terraform modules

There is enough documentation regarding the creation of Terraform modules and I am going to focus mostly on our approach towards consumption of modules across multiple teams.

See the examples on Github. A very simple module can look like

variable "name" {
  type = string
}

variable "location" {
  type = string
}

variable "tags" {
  type = map
}

resource "azurerm_resource_group" "resource_group" {
  name     = var.name
  location = var.location
  tags = var.tags
}

output "name" {
  value = azurerm_resource_group.resource_group.name
}

And the way to consume it will be

provider "azurerm" {
    features {}
}

locals {
  tags = {
    "key1" = "value1"
  }
}

module "resource_group" {
  source = "git::https://github.com/realrubberduckdev/tf-module-test.git//modules//ResourceGroup?ref=9451c9a386b411b71400aa1a382c3e93b2b9e9a0"
  name     = "tfmoduletest3"
  location = "North Europe"
}

module "storage_account" {
  source = "git::https://github.com/realrubberduckdev/tf-module-test.git//modules//StorageAccount?ref=9451c9a386b411b71400aa1a382c3e93b2b9e9a0"
  name     = "dpteststrgacc1"
  resource_group_name = module.resource_group.name
  location = "North Europe"
}

Notice this in main.tf on the GitHub repo. The git as module source lets us use a git repo we have access to while locking down to a specific commit hash. So product teams can use these modules in their repo while ensuring, they do not automatically pick up breaking changes.

Possibilities

Semver: The Platform team can use tools like GitVersion to adopt semantic versioning and automate the module versioning using git tags. So there references do not need to be an unreadable hash, rather can be ref=v1.0.0.
Templates: With proper cost implication investigations, the Platform team can group resources into being resource templates. Thus reducing overall cost for the organization.
Dotnet tool: With a simple dotnet tool, we can adopt an organization-wide convention regarding state file management.

Conclusion

This post is more about the concept and the workflow model and how Terraform helps us with the model adoption. Surely there will be other better technologies for IaC and the quest to find a better one shall continue. Hopefully the one without state file management (more on this in a separate post). I hope this was useful, please do share any thoughts or comments you might have.

Cosmos DB TransactionalBatch - Dealing with BadRequest StatusCode

Sat, 27 Mar 2021 12:30:00 GMT

Introduction

Azure Cosmos DB provides a great way to perform transactional batch operations using the .Net SDK. This operation method provides the ACID guarantees and so it makes it much easier to perform multiple steps in one transactional batch or fail and roll back the whole operation.

Dealing with BadRequest StatusCode

The Microsoft documentation link above does a fantastic job of explaining how to implement a transactional batch operation, so won’t go much into that. But when I was trying it out, I kept getting StatusCode=BadRequest and it can be quite difficult to find out why that happens.

I have been unable to find out details from the response object.

var result = response.GetOperationResultAtIndex<Customer>(0);

Even the OperationResult wasn’t helpful, check the image below. Note that, Customer is the object I am trying to write to the database.

Say we have a function GetResponse like below:

Finding out the issue

The only way I could find out what the issue was by completely ditching the TransactionalBatch method and test it using

await container.CreateItemAsync<Customer>(customerItem, new PartitionKey(customerItem.Id));

And this now led me to another exception, which was a bit easier to understand.

PartitionKey extracted from document doesn't match the one specified in the header on CreateItemAsync

Explanation

So if you have followed the Microsoft documentation, it lists two limitations. The bit that it doesn’t list, which seems to be by design is a TransactionalBatch can only work in a single Partition.

TransactionalBatch batch = container.CreateTransactionalBatch(new PartitionKey(customer.PartitionKey))

So when we try to get the container, we have to specify the PartitionKey too as the second argument.

var container = await database.CreateContainerIfNotExistsAsync("CustomerInfo", "/id");

And both the partition keys need to be the same, otherwise TransactionalBatch will keep throwing a BadRequest.

Solution

This stackoverflow post touches on the solution. So overall we need to ensure our Customer class has a PartitionKey which Cosmos DB can serialize.

public class Customer
  {
    [JsonProperty(PropertyName = "id")] // This is needed by Cosmos DB
    public string? Id { get; set; }

    [JsonProperty(PropertyName = "CustomerPartition")] // This is needed by Cosmos DB. Note that it cannot take any special characters, not even hyphen.
    public string? CosmosPartitionKey { get; set; } = "CustomerPartition";

  }

Along with this we also need to ensure the Container, as well as TransactionalBatch, are pointing to the same PartitionKey.

string partitionKey = "CustomerPartition";
var container = await database.CreateContainerIfNotExistsAsync("CustomerInfo", $"/{partitionKey}");
TransactionalBatch batch = container.CreateTransactionalBatch(new PartitionKey(partitionKey))

Conclusion

Hope this was useful and saves you some time if you are trying this out. If you have any thoughts or comments please do get in touch with me on Twitter @rubberduckdev. Or use the Disqus plugin below.

Mocking HttpClient for unit testing

Tue, 19 Jan 2021 18:30:00 GMT

Introduction

HttpClient is an easy way provided by dotnet to send http requests. But the problem is that it doesn’t implement an interface we could mock for unit testing.

How to mock HttpClient

Say we have a function GetResponse like below:

public async Task<string?> GetResponse(HttpClient httpClient, string requestUri)
{
        var request = new HttpRequestMessage
        {
            Method = HttpMethod.Get,
            RequestUri = new Uri(requestUri)
        };

        using var response = await httpClient.SendAsync(request);
        response.EnsureSuccessStatusCode();
        string? responseBody = await response.Content.ReadAsStringAsync();
        return responseBody;
}

Moq for mocking

Let’s use Moq for mocking. If you notice the method we would like to stub is SendAsync, which is a protected method in HttpClient class. The way to mock that is:

public static HttpClient GetMockedHttpClient(string responseContent)
{
    var handlerMock = new Mock<HttpMessageHandler>();
    var response = new HttpResponseMessage
    {
        StatusCode = HttpStatusCode.OK,
        Content = new StringContent(responseContent),
    };

    handlerMock
        .Protected()
        .Setup<Task<HttpResponseMessage>>(
            "SendAsync",
            ItExpr.IsAny<HttpRequestMessage>(),
            ItExpr.IsAny<CancellationToken>())
        .ReturnsAsync(response);
    var httpClient = new HttpClient(handlerMock.Object);
    return httpClient;
}

Notice how the SendAsync method has now been stubbed and this can be used in unit tests.

So in a unit test we can call GetResponse as follows:

var mockedHttpClient = GetMockedHttpClient("{}");
var response = someObject.GetResponse(mockedHttpClient, "some-uri");

Conclusion

Hope this was useful and saves you some time if you are trying this out. If you have any thoughts or comments please do get in touch with me on Twitter @rubberduckdev. Or use the Disqus plugin below.

Azure project during the holidays of Dec 2020

Fri, 01 Jan 2021 08:30:00 GMT

Introduction

In recent holidays at the end of 2020, I thought of brushing up my dotnet core web development skills or to be honest learn about it. I have never been a professional web developer. My UI development skills still suck, but I learnt quite a bit about the whole architecture which is what I plan to share in this post.

The idea

The overall idea is to create an online service which can tell us where a certain tv show or movie is streaming. So, for example, we can search for Tenet and it will find it for us where it is streaming, say Amazon prime for example. I managed to deploy the website with a free domain, thanks to freenom. Registration and binding of a custom domain with the azure app service can be a post by itself, maybe one for later.

Architecture diagram

Planning to share the overall architecture of the project, which is as follows:

The main architecture bit is the one which is part of the project. There are two parts, the website and an android app which is under development at the moment. Both of them depend on a dotnet core class library which handles all the logic. It calls into Tmdb API and UTelly API via RapidAPI. Tmdb is free and allows unlimited calls, so that is good for this project. But RapidApi only allows 1000 free calls a month.

To reduce operational costs, I used Azure Cache for Redis. The call to Tmdb has a TTL of 1 day (to avoid redundant calls) and the Rapid API one has a TTL of 30 days. So we can do 1000 unique free calls, anything more than that will have a cost.

CI CD

In my opinion, a good, reliable CI-CD is a must for robust software development. Azure DevOps (ADOS from here on) provides a very good suite of tools for this. I like GitHub actions, but as most of my private projects are currently on ADOS, it is easier for me to leverage that.

The whole continuous integration process of build, test and the following continuous deployment is automated. I am in the process of adding one manual approval step before the staging-production slot gets swapped.

Conclusion

In conclusion, if you are curious about where your favourite show is streaming, do check it out at https://wherestreaming.online.

Overall doing this project during the holidays was both fun as well interesting. Learnt many tips and tricks which I will share in follow up posts. If you have any thoughts or comments please do get in touch with me on Twitter @rubberduckdev. Or use the Disqus plugin below.

Static analysis of Azure resource configuration

Thu, 24 Dec 2020 08:30:00 GMT

Introduction

Static code analysis of code has been there for quite a while now and it is very useful now in case of configuration as code scenario as well. Often, we make mistakes in our configuration code (HCL or not), but if you are using Terraform HCL then TFSec is a good choice. When I tried, I was up and running within 5 to 10 minutes, soon followed by CI with Github actions. I like docker in general, so took the docker approach, which hopefully makes it more usable across environments.

Festive Tech Calendar 2020

This demo is designed as my contribution for Festive Tech Calendar 2020, definitely check out the website for some many other great topics. Happy Christmas.

This video is part of the festive tech calendar 2020.

GitHub

All the code shown in the video is available at https://github.com/realrubberduckdev/terraform-static-analysis.

Conclusion

Hope this was useful and saves you some time if you are trying this out. Please do share your learnings. If you have any thoughts or comments please do get in touch with me on Twitter @rubberduckdev. Or use the Disqus plugin below.

Xamarin - Complying with AdMob policies

Sat, 31 Oct 2020 18:30:00 GMT

Introduction

Xamarin has made monetization of mobile cross-platform apps very easy. Mainly with nuget packages such as Xamarin.GooglePlayServices.Ads.Lite, adding admob banner ads has become a breeze.

The Xamarin Show

James Montemagno has a great video on the Xamarin show explaining how to add banner ads on android apps.

SimpleJigsaw

With help and inspiration from James Montemagno, two of my close friends Sanjeeb Swain & Suryakiran Maruvada and I set out to design our first android app.

Monetizing SimpleJigsaw

Following the video above and a few other tutorials, we managed to add ads in our app.

Obviously, we got AdMob id details with a banner unit:
And then added these along with the ad control view, ad view renderer and other updates in a commit. This is available on github for reference.

AdMob Policy Issue

Although the commit above works perfectly on the local machine and for a day or so did work in the published app, there was an issue. Within a few days, I received an email from Google.

On visiting the policy centre, we found the following message:

Modified ad code: Resizing Ad Frames
MODIFIED ADS: Publishers are not permitted to alter the behaviour of Google ads in any way. This includes resizing ad frames to cut off parts of ads or hiding the Ads by Google moniker.

Quite baffling for untrained and hobbyist Xamarin developers. Because violating policy was never the intention. Then we started following the policies in more detail at Google AdMob help. Finally, we understood that the ad banner under no circumstances should be clipped/resized and that can be avoided by ensuring it gets the height it gets.

To achieve this we added IsClippedToBounds="False" to the grid view containing the ad banner.

<Grid IsClippedToBounds="False">
    <Grid.ColumnDefinitions>
        <ColumnDefinition Width="*" />
    </Grid.ColumnDefinitions>
    <Grid.RowDefinitions>
        <RowDefinition Height="Auto" />
    </Grid.RowDefinitions>
    <local:AdControlView x:Name="adControlView" Grid.Row="0"/>
</Grid>

And in the next bit was ensure it gets the size it wants. The smart banner sizing is described at Google AdMob help.

    class AdViewRenderer : ViewRenderer
    {
        private Context _context;

        public AdViewRenderer(Context context) : base(context)
        {
            _context = context;
        }

        protected override void OnElementChanged(ElementChangedEventArgs<View> e)
        {
            base.OnElementChanged(e);
            if (Control == null)
            {
                var ad = new AdView(_context);
                ad.AdSize = AdSize.SmartBanner;
                ad.AdUnitId = SimpleJigsawConstants.adMobBannerUnitId;
                var requestbuilder = new AdRequest.Builder();
                ad.LoadAd(requestbuilder.Build());
                e.NewElement.HeightRequest = GetSmartBannerDpHeight();

                SetNativeControl(ad);
            }
        }

        private int GetSmartBannerDpHeight()
        {
            var dpHeight = Resources.DisplayMetrics.HeightPixels / Resources.DisplayMetrics.Density;

            if (dpHeight <= 400) return 32;
            if (dpHeight > 400 && dpHeight <= 720) return 50;
            return 90;
        }
    }

In order to obtain the correct size of banner, we introduced a new function GetSmartBannerDpHeight() in the AdViewRenderer class. This would automatically get the correct height as that is dynamic.

One more thing to note is that in the XAML file, we started naming the adControlView:

<local:AdControlView x:Name="adControlView" Grid.Row="0"/>

This is because we learnt from a community post that if an ad banner is active, the user must not be restricted to click on it, else it will cause a policy violation. So we use that name in a wrapper function to ensure the banner gets hidden if we are to show an alert.

private async Task<bool> PerformCustomDisplayAlert(string title, string message, string accept, string cancel)
{
    // https://www.b4x.com/android/forum/threads/solved-google-admob-restricted-ad-serving-modified-ad-code-resizing-ad-frames.112884/
    // Hide ad before showing alert, else Google thinks we are blocking user access to their ads.
    adControlView.IsVisible = false;
    var res = await DisplayAlert(title, message, accept, cancel);
    adControlView.IsVisible = true;
    return res;
}

And since these changes, our ad seems to work fine on our app. The full demo code is available on github.

Conclusion

As you may have already got, this was a learning experience. And this development is as a hobby, we welcome any help and advise you can provide. Definitely try out the app if you can and provide feedback.

Terraform - Raise an error

Thu, 17 Sep 2020 08:30:00 GMT

Introduction

The ability to conditionally raise errors is very useful if we are introducing logic and rules in our deployment. Say for example we need to validate input then we can use the newly introduced variable validation feature in Terraform 0.13. But there could be a specific case where you’d want to raise an error or you’ve used a workaround for this validation in 0.11 which no longer works after the upgrade. More on this in a bit, and that is the case I hit recently.

Workaround

In Terraform 0.11, there was no ability to validate input variables. So we used a workaround with null_resource

resource "null_resource" "throw_error" {
      count = 1
      "An error has occured." = true
}

But unfortunately and possibly rightly so, HCL became stricter from 0.12 onwards and would not allow incorrect syntax such as "string" = true and fail. There is a stackoverflow question regarding this. Along with the StackOverflow question, there are lots of requests and suggested workarounds on GitHub such as Ability to raise an error & conditionally raise a configuration error.

So the workaround we went for in my my team, suggested on github is:

data "external" "throw_error" {
    count = 1 # 0 means no error is thrown, else throw error
    program = ["powershell.exe", "throw 'An error has ocurred.'"]
}

Explanation

The workaround uses external data source which allows an external program to act as a data source. So essentially it is a hack, saying PowerShell will be our data source but instead, it throws an error. This happens conditionally only if we set the count to greater than 0. If set to 0, this data block is not executed.

Conclusion

Hope this was useful and saves you some time. Please do share your learnings. If you have any thoughts or comments please do get in touch with me on twitter @rubberduckdev. Or use the Disqus plugin below.

Why is my PowerShell function returning two values?

Wed, 05 Aug 2020 17:30:00 GMT

Introduction

This post is regarding a small PowerShell function that I was writing. It started returning two values at the same time and it took me a while to figure out what was happening. Hence this post and hope it saves someone else some time.

The function

The objective of the function we are writing is to check if an item is present in a JSON array.

{
    [
        {
            "displayName": "Apple",
            "description": "Fruit"
        },
        {
            "displayName": "Potato",
            "description": "Vegetable"
        }
    ]
}

Our JSON array is a simple displayName & description pair elements. And we need an ItemExists function in PowerShell returning boolean value, true if the item exists and false if it doesn’t.

Here is a script with the function:

function ItemExists {
    param (
        [PSCustomObject]$jsonData,
        [string]$itemToLookFor
    )
    $jsonData | Select-Object -Property displayName | ForEach-Object {
        if ($itemToLookFor -eq $_.displayName) {
            return $true
        }
    }
    return $false
}

$json = @"
    [
    {
        "displayName": "Apple",
        "description": "Fruit"
    },
    {
        "displayName": "Potato",
        "description": "Vegetable"
    }
    ]
"@ | ConvertFrom-Json

$exists = ItemExists -jsonData $json -itemToLookFor "Apple"
Write-Host exists=$exists

As you can see, we are looking for string Apple and it exists, so it should return true. But the output is as follows:

exists=True False

Is the function returning two values? And if yes how and if no then why is this happening!

Explanation

The point of interest here is ForEach-Object. Beginning with PowerShell 7.0, it runs each script block in parallel. The first return statement is within the scope of the ForEach-Object and the second one is outside of the scope. Meaning the first return is returning from ForEach-Object which is in a separate runspace. This then leads to the next return statement. So the function is returning two values, due to the loop running in separate runspaces.

Rewriting the function

How do we get the function to work then? Let’s rewrite the function without ForEach-Object, rather we can use foreach loop, as follows.

function ItemExists {
    param (
        [PSCustomObject]$jsonData,
        [string]$itemToLookFor
    )
    $items = $jsonData | Select-Object -Property displayName
    foreach ($item in $items) {
        if ($itemToLookFor -eq $item.displayName) {
            return $true
        }
    }
    return $false
}

$json = @"
    [
    {
        "displayName": "Apple",
        "description": "Fruit"
    },
    {
        "displayName": "Potato",
        "description": "Vegetable"
    }
    ]
"@ | ConvertFrom-Json

$exists = ItemExists -jsonData $json -itemToLookFor "Apple"
Write-Host exists=$exists

This time the loop doesn’t run in separate runspaces and hence the output is:

exists=True

This is exactly the value we wanted.

Conclusion

Finding out IPs of Azure function

Sun, 02 Aug 2020 15:00:00 GMT

Introduction

The IP address of an Azure function can be very useful, especially if you are using IP based network restrictions or any kind of firewall to block requests from unknown IPs. Azure gives us a single inbound IP address to the function, but a range of possible outbound ones. This is so that Azure can do its internal load balancing. The issue you might hit is that these IPs can all change.

When can IP change

There are specific cases when IPs can change. Again these are listed in the documentation. Both the inbound and outbound IP addresses can change. So how do we figure out, on the fly, what are the IPs of our Azure function?

Resource graph

This is when Azure Resource Graph becomes super useful. The resource graph is a powerful management tool to query, explore and analyse your cloud resources at scale. We can see all resources, their properties etc using Kusto query language (KQL) queries.

Using resource graph

Steps to finding out the IPs of our function are as follows. Note that we have a function app named blog-test-funap.

Go to Azure Resource Graph Explorer on your Azure portal

Enter query

Resources | where name =~ 'blog-test-funap' and type =~ 'microsoft.web/sites'

Notice the =~ which means, do a case insensitive comparison. And we filter the resources down to microsoft.web/sites, which is what function apps are under the hood.

Scroll and click on See details
See possible IP addresses in Properties

Note that there are multiple JSON properties to get full list of IP addresses such as possibleOutboundIpAddresses, possibleInboundIpAddresses and outboundIpAddresses.

Conclusion

Azure resource graph gives a lot of power at your fingertips. This post was a demonstration at Azure portal, but we can do the same using AZ CLI, Azure Powershell or dotnet sdk or REST API. Thereby having full knowledge of our Azure resources with automation.

Hope this was useful. Please do share your learnings. If you have any thoughts or comments please do get in touch with me on twitter @rubberduckdev. Or use the Disqus plugin below.

Access restrictions in Azure functions

Sun, 02 Aug 2020 12:00:00 GMT

Introduction

Secured access to azure functions is very important, especially when your distributed application can have a public-facing infrastructure. Be it public IP addresses or user interface. Azure RBAC roles provide a great way to control who can trigger your function or what your function can do. On top of that, network access restrictions add the perfect layer of security to avoid unwanted sources to be able to trigger your function.

Developer access

During the development phase, any access restrictions set up can cause issues for the developers. For example, if we create a sample function and add a restriction rule (check screenshot), then a Deny all is automatically added at the end of the restrictions list. The logic being, if we want specific IPs/Vnets etc to be allowed in, then we must want the rest of the sources to be locked out.

It makes sense. But the issue is as a developer, you can no longer test run the function on your azure portal.

As the source of the trigger, the Azure portal is now from your developer machine. So you see the following error:

You must have direct network access in order to run your function. Your app may be restricted with Private Endpoints, Access Restrictions or Service Endpoints.

Allow list yourself

As we know why our request is being blocked, the simplest solution is to allow list ourself.

Steps:

Get your public ip, I used google
Add to network restrictions

Ensure you use the CIDR to allow list only the specific IP and not a range. For security, you do not want more than your IP getting the access to your function.

If the concept is new, here is a fantastic video explaining how the subnetting is calculated using CIDR notation.

This is how it should look after the rule is added.

Note that you may have to wait few minutes for the rule to propagate. Then trigger the function from the portal:

And that’s it, now you should be able to trigger your function.

Quick detour to wording conventions

As a quick detour, the tech industry has recently started using more explicit terminology and removing old wordings which both do not make complete sense and can be offensive. Hence the term, allow list. It will be a slow change, let’s try.

Remove before production

Once your development phase is done, definitely remove any unwanted IPs you have added. I normally recommend having a unit test which can use REST API to assert what rules should be present, before the function is deployed to production.

Conclusion

Hope this was useful. I am slowly learning new techniques to work with Azure functions. Please do share your learnings. If you have any thoughts or comments please do get in touch with me on twitter @rubberduckdev. Or use the Disqus plugin below.

Debugging an Event Grid Triggered Azure Function

Fri, 26 Jun 2020 19:00:00 GMT

Event grid trigger Azure function

We can use Azure Event Grid with an Azure function as an endpoint to handle the events. With great development tools available for Azure functions with Visual Studio, we can debug into our function with a real/mock event grid event trigger.

This post needs an understanding of the basics of Azure functions development and event grid.

Debugging with a real event from Azure

For this technique, we need to allow Azure to be able to call the function which we have on our local machine. The steps to follow are:

Start debugging locally in Visual Studio (hit F5).
Note the port number from the debug console. It normally is 7071.
Now we need to allow Azure to be able to call our function. For this, we use the ngrok utility. Get the utility then run the command
```
ngrok http -host-header=localhost 7071
```
From the ngrok run output, copy the https URL.

We then use this on our Azure event grid subscription endpoint in the lines of

https://aeccb7806acc.ngrok.io/runtime/webhooks/EventGrid?functionName=Function1

And that’s it. When there is a new event, ngrok will redirect the endpoint call to the function on our development machine.

Debugging locally with mock events

For locally creating mocked up events, we need to understand the schema of the event.

Creating the event JSON

Check the event schema on Microsoft documentation.

[
  {
    "topic": string,
    "subject": string,
    "id": string,
    "eventType": string,
    "eventTime": string,
    "data":{
      object-unique-to-each-publisher
    },
    "dataVersion": string,
    "metadataVersion": string
  }
]

Event data

The data element varies depending on the different event sources. Say for example we want to mock the events from Azure subscription as an event grid source. An example event can be found in Microsoft documentation.

It looks like

[{
  "subject": "/subscriptions/{subscription-id}/resourcegroups/{resource-group}/providers/Microsoft.Storage/storageAccounts/{storage-name}",
  "eventType": "Microsoft.Resources.ResourceWriteSuccess",
  "eventTime": "2018-07-19T18:38:04.6117357Z",
  "id": "4db48cba-50a2-455a-93b4-de41a3b5b7f6",
  "data": {
    "authorization": {
      "scope": "/subscriptions/{subscription-id}/resourcegroups/{resource-group}/providers/Microsoft.Storage/storageAccounts/{storage-name}",
      "action": "Microsoft.Storage/storageAccounts/write",
      "evidence": {
        "role": "Subscription Admin"
      }
    },
    "claims": {
      "aud": "{audience-claim}",
      "iss": "{issuer-claim}",
      "iat": "{issued-at-claim}",
      "nbf": "{not-before-claim}",
      "exp": "{expiration-claim}",
      "_claim_names": "{\"groups\":\"src1\"}",
      "_claim_sources": "{\"src1\":{\"endpoint\":\"{URI}\"}}",
      "http://schemas.microsoft.com/claims/authnclassreference": "1",
      "aio": "{token}",
      "http://schemas.microsoft.com/claims/authnmethodsreferences": "rsa,mfa",
      "appid": "{ID}",
      "appidacr": "2",
      "http://schemas.microsoft.com/2012/01/devicecontext/claims/identifier": "{ID}",
      "e_exp": "{expiration}",
      "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname": "{last-name}",
      "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname": "{first-name}",
      "ipaddr": "{IP-address}",
      "name": "{full-name}",
      "http://schemas.microsoft.com/identity/claims/objectidentifier": "{ID}",
      "onprem_sid": "{ID}",
      "puid": "{ID}",
      "http://schemas.microsoft.com/identity/claims/scope": "user_impersonation",
      "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier": "{ID}",
      "http://schemas.microsoft.com/identity/claims/tenantid": "{ID}",
      "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name": "{user-name}",
      "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn": "{user-name}",
      "uti": "{ID}",
      "ver": "1.0"
    },
    "correlationId": "{ID}",
    "resourceProvider": "Microsoft.Storage",
    "resourceUri": "/subscriptions/{subscription-id}/resourcegroups/{resource-group}/providers/Microsoft.Storage/storageAccounts/{storage-name}",
    "operationName": "Microsoft.Storage/storageAccounts/write",
    "status": "Succeeded",
    "subscriptionId": "{subscription-id}",
    "tenantId": "{tenant-id}"
  },
  "dataVersion": "2",
  "metadataVersion": "1",
  "topic": "/subscriptions/{subscription-id}"
}]

Final event JSON

We have to merge the above two JSON files to form the final one which should look like

{
    "subject": "/subscriptions/0not-real-subcription-id1/resourcegroups/resource-group-name/providers/Microsoft.Web/sites/webappname/config/appsettings",
    "eventType": "Microsoft.Resources.ResourceWriteSuccess",
    "eventTime": "2018-07-19T18:38:04.6117357Z",
    "id": "4db48cba-50a2-455a-93w4-de41a3b5b7f6",
    "data": {
        "authorization": {
            "scope": "/subscriptions/0not-real-subcription-id1/resourceGroups/resource-group-name/providers/Microsoft.Web/sites/webappname/config/appsettings",
            "action": "Microsoft.Web/sites/config/write",
            "evidence": {
                "role": "Contributor",
                "roleAssignmentScope": "/subscriptions/0not-real-subcription-id1",
                "roleAssignmentId": "4885d9f5deb94e8d9008e0489072b146",
                "roleDefinitionId": "b24988ac618042a0ab8820f7382dd24c",
                "principalId": "8c80ef36f16f4bd4b237799e24c43cb5",
                "principalType": "Group"
            }
        },
        "claims": {
            "aud": "https://management.core.windows.net/",
            "iss": "https://sts.windows.net/49d1-49bc-b63c/",
            "iat": "1593077152",
            "nbf": "1593077152",
            "exp": "1593081052",
            "http://schemas.microsoft.com/claims/authnclassreference": "1",
            "aio": "AUQAu/sdfdsf+5Bzb5OcRKrcNyHu5w==",
            "http://schemas.microsoft.com/claims/authnmethodsreferences": "pwd,rsa,mfa",
            "appid": "1950a258-227b-4e31-a9cf-717495945fc2",
            "appidacr": "0",
            "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname": "Priyadarshee",
            "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname": "Dushyant",
            "groups": "76e57378-7986-491e-sdf-6220d6a8bb13",
            "ipaddr": "193.117.214.98",
            "name": "Priyadarshee, Dushyant",
            "http://schemas.microsoft.com/identity/claims/objectidentifier": "f0dc630a-b9b9-498a-945d-5c86dd3a9cbc",
            "onprem_sid": "S-1-5-21-1102854320-3722712898-1019641694-40955",
            "puid": "10032000C1028AF8",
            "http://schemas.microsoft.com/identity/claims/scope": "user_impersonation",
            "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier": "RctV-GGTWjD_kITSHLSCIJJD4hwLLdO4o7J7Vi5NT5Q",
            "http://schemas.microsoft.com/identity/claims/tenantid": "49d1-49bc-b63c",
            "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name": "abcdef@abc.def.co.uk",
            "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn": "abcdef@abc.def.co.uk",
            "uti": "fPTvpJqJ-kib1Ui5IfnqAA",
            "ver": "1.0",
            "wids": "88d8e3e3-8f55-4a1e-953a-9b9898b8876b"
        },
        "correlationId": "92db1c00-aa0d-4044-80b0-de64673e1295",
        "httpRequest": {
            "clientRequestId": "3da63058-b6b6-4f1e-a6aa-28420bab4594",
            "clientIpAddress": "193.17.210.98",
            "method": "PUT",
            "url": "https://management.azure.com/subscriptions/0not-real-subcription-id1/resourceGroups/resource-group-name/providers/Microsoft.Web/sites/webappname/config/appsettings?api-version=2018-11-01"
        },
        "resourceProvider": "Microsoft.Web",
        "resourceUri": "/subscriptions/0not-real-subcription-id1/resourceGroups/resource-group-name/providers/Microsoft.Web/sites/webappname/config/appsettings",
        "operationName": "Microsoft.Web/sites/config/write",
        "status": "Succeeded",
        "subscriptionId": "0not-real-subcription-id1",
        "tenantId": "49d1-49bc-b63c"
    },
    "dataVersion": "2",
    "metadataVersion": "1",
    "topic": "/subscriptions/0not-real-subcription-id1"
}

We would want to replace the values in the JSON as to what our function is likely to expect.

Using postman to create the request

Now using postman we create the POST request.

We use the URL http://localhost:7071/runtime/webhooks/EventGrid?functionName=Function1
Add the following headers
```
Content-Type = application/json
aeg-event-type = Notification
```
These headers are discussed briefly in Event hub as an event handler for Azure Event Grid events.
We copy paste the json we created earlier into request body

And that’s it, when we click Send, we will hit any breakpoints on our Visual Studio instance.

Conclusion

This has been a learning experience for me and hence considered sharing if it is useful for someone else. I must say the documentaion provided by Microsoft is very useful in learning these among other tricks. There is also a great blog post by Paul Mcilreavy regarding this.

Overall, these techniques make debugging experience so much easier and we have all Visual Studio tools at our disposal in these scenarios as well. Definite win-win.

Event grid Azure subscription filtering

Tue, 23 Jun 2020 19:00:00 GMT

Event grid

Azure Event Grid is designed to build applications with event-based architectures. The events can be published from Azure or other sources and can be handled by services on Azure or elsewhere as long as they can subscribe to them. Event Grid has built-in support for events coming from Azure services, like storage blobs and resource groups. We can also design custom topics for our own events to be published onto the event grid.

Event grid concepts

The key concepts of the event grid are in the diagram below.

Essentially, picked straight from documentation. There are five concepts in Azure Event Grid that let you get going:

Events - What happened.
Event sources - Where the event took place.
Topics - The endpoint where publishers send events.
Event subscriptions - The endpoint or built-in mechanism to route events, sometimes to more than one handler. Subscriptions are also used by handlers to intelligently filter incoming events.
Event handlers - The app or service reacting to the event.

Subscribing to Azure subscription topic

So in this post, we will focus on subscribing to events such as changes that happen to resources on Azure. This could be a file uploaded to blob storage, a new azure function created or something else. The main trick to getting the specific event we want is by using filters.

Azure portal

I would not write on this part, as lots of resources are available online including documentation. Providing it here, for the sake of providing the easiest way to try and test.

ARM Template

ARM templates provide a great means to implement IaC and this is the main bit we will focus on in this post.

Use case

Let’s try to filter events in event grid to be triggered only when an Azure function changes in our subscription (say configuration change, code updated or new function created/deleted).

Steps to put together the template

Get the quick start template from 101-event-grid-resource-events-to-webhook.

Update filters in arm template

            "filter": {
                "isSubjectCaseSensitive": false,
                "subjectBeginsWith": "",
                "subjectEndsWith": "",
                "includedEventTypes": [
                    "Microsoft.Resources.ResourceWriteSuccess",
                    "Microsoft.Resources.ResourceDeleteSuccess"
                ],
                "advancedFilters": [
                    {
                        "values": [
                            "Microsoft.Web/sites/functions/write",
                            "Microsoft.Web/sites/functions/delete",
                            "Microsoft.Web/sites/hostruntime/vfs/run.csx/write",
                            "Microsoft.Web/sites/config/write"
                        ],
                        "operatorType": "StringIn",
                        "key": "data.operationName"
                    }
                ]
            }

As far as I understand all values in filter section excluding advancedFilters are mandatory, at least how VSCode directed me. For the sake of our use case, we only need to look at Microsoft.Resources.ResourceWriteSuccess & Microsoft.Resources.ResourceDeleteSuccess events. The full list of event types for Azure subscription topic can be found at Event Grid event schema. Please note that the Azure subscription is our subscription on Azure and it is not implying subscribing the event grid.

After getting the right type of events, we would want to filter it down to specific changes in Azure functions. This can be done via advancedFilters. First of all, we need to look at an example event from Azure subscription.

[{
  "subject": "/subscriptions/{subscription-id}/resourcegroups/{resource-group}/providers/Microsoft.Storage/storageAccounts/{storage-name}",
  "eventType": "Microsoft.Resources.ResourceWriteSuccess",
  "eventTime": "2018-07-19T18:38:04.6117357Z",
  "id": "4db48cba-50a2-455a-93b4-de41a3b5b7f6",
  "data": {
    "authorization": {
      "scope": "/subscriptions/{subscription-id}/resourcegroups/{resource-group}/providers/Microsoft.Storage/storageAccounts/{storage-name}",
      "action": "Microsoft.Storage/storageAccounts/write",
      "evidence": {
        "role": "Subscription Admin"
      }
    },
    "claims": {
      "aud": "{audience-claim}",
      "iss": "{issuer-claim}",
      "iat": "{issued-at-claim}",
      "nbf": "{not-before-claim}",
      "exp": "{expiration-claim}",
      "_claim_names": "{\"groups\":\"src1\"}",
      "_claim_sources": "{\"src1\":{\"endpoint\":\"{URI}\"}}",
      "http://schemas.microsoft.com/claims/authnclassreference": "1",
      "aio": "{token}",
      "http://schemas.microsoft.com/claims/authnmethodsreferences": "rsa,mfa",
      "appid": "{ID}",
      "appidacr": "2",
      "http://schemas.microsoft.com/2012/01/devicecontext/claims/identifier": "{ID}",
      "e_exp": "{expiration}",
      "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname": "{last-name}",
      "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname": "{first-name}",
      "ipaddr": "{IP-address}",
      "name": "{full-name}",
      "http://schemas.microsoft.com/identity/claims/objectidentifier": "{ID}",
      "onprem_sid": "{ID}",
      "puid": "{ID}",
      "http://schemas.microsoft.com/identity/claims/scope": "user_impersonation",
      "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier": "{ID}",
      "http://schemas.microsoft.com/identity/claims/tenantid": "{ID}",
      "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name": "{user-name}",
      "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn": "{user-name}",
      "uti": "{ID}",
      "ver": "1.0"
    },
    "correlationId": "{ID}",
    "resourceProvider": "Microsoft.Storage",
    "resourceUri": "/subscriptions/{subscription-id}/resourcegroups/{resource-group}/providers/Microsoft.Storage/storageAccounts/{storage-name}",
    "operationName": "Microsoft.Storage/storageAccounts/write",
    "status": "Succeeded",
    "subscriptionId": "{subscription-id}",
    "tenantId": "{tenant-id}"
  },
  "dataVersion": "2",
  "metadataVersion": "1",
  "topic": "/subscriptions/{subscription-id}"
}]

In the example event JSON, notice operationName. The full list of available operations can be found at Azure resource providers operations. For our use case, we only need to look at specific operations from that list. And those values can be accessed via the data object as in our template.

                    "advancedFilters": [
                        {
                            "values": [
                                "Microsoft.Web/sites/functions/write",
                                "Microsoft.Web/sites/functions/delete",
                                "Microsoft.Web/sites/hostruntime/vfs/run.csx/write",
                                "Microsoft.Web/sites/config/write"
                            ],
                            "operatorType": "StringIn",
                            "key": "data.operationName"
                        }
                    ]

Final ARM template should look like the one below:

{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
    "eventSubName": {
        "type": "string",
        "defaultValue": "subToResources",
        "metadata": {
            "description": "The name of the event subscription to create."
        }
    },
    "endpoint": {
        "type": "string",
        "metadata": {
            "description": "The URL for the WebHook to receive events. Create your own endpoint for events."
        }
    }
},
"resources": [
    {
        "type": "Microsoft.EventGrid/eventSubscriptions",
        "name": "[parameters('eventSubName')]",
        "apiVersion": "2018-01-01",
        "properties": {
            "destination": {
                "endpointType": "WebHook",
                "properties": {
                    "endpointUrl": "[parameters('endpoint')]"
                }
            },
            "filter": {
                "isSubjectCaseSensitive": false,
                "subjectBeginsWith": "",
                "subjectEndsWith": "",
                "includedEventTypes": [
                    "Microsoft.Resources.ResourceWriteSuccess",
                    "Microsoft.Resources.ResourceDeleteSuccess"
                ],
                "advancedFilters": [
                    {
                        "values": [
                            "Microsoft.Web/sites/functions/write",
                            "Microsoft.Web/sites/functions/delete",
                            "Microsoft.Web/sites/hostruntime/vfs/run.csx/write",
                            "Microsoft.Web/sites/config/write"
                        ],
                        "operatorType": "StringIn",
                        "key": "data.operationName"
                    }
                ]
            }
        }
    }
]
}

Create an Azure function with event grid trigger. This function must exist before we deploy the arm template.
Deploy arm template with endpoint parameter pointing to the function URL we created. Ensure they have the right keys or the handshaking will fail.

Credits

I must mention these two gentlemen who helped me while I was trying to figure out how best to create the arm template. Dan Clarke for the initial tip to start investigating towards event grid for this use case and Roman Kiss for answering my question on stackoverflow and helping me out with the filters.

Conclusion

It has been a satisfying learning experience figuring out the best means to trigger endpoints with events coming from resources. Event grid makes this very useful and easy. The possibilities are endless, we can have functions and services which act as per behaviour from resources. It makes automation of Azure resources easier too. Hope this post helps you in your efforts with event grid and arm templates. Do share thoughts and opinions on twitter or here.

Obfuscation using nuget

Thu, 04 Jun 2020 19:00:00 GMT

Obfuscation

We are in the golden age of open source software and yet at least I (surely many others too) work at places where safeguarding intellectual property (IP) in software is of paramount importance. One of the main techniques used to safeguard IP is by obfuscating the code we ship. We can achieve this by a simple nuget as well. Let’s dig into the details in this post.

Nuget

This post mainly aims at .net framework project, msbuild and nuget.

.Net framework

You may think why are we still talking about .net framework when .NET Core is the Future of .NET & .NET 5 is round the corner. We are getting cool features with the latest developments in the dotnet world, but again from experience, we know that migration and adoption of the newer technologies will take some time, I mean years. Until then we will have a codebase of only .net framework or a mix of .net framework and dotnet core/standard etc.

Having said so, the sdk style csproj is very useful, feel free to check my earlier blog post regarding that. But as of now, we do not have an sdk style csproj template shipped in Visual Studio for .net framework projects. I use the simple visual studio extension above, which is open source and any comments/suggestions are welcome on that.

NuGet techniques

Before we jump into the obfuscation nuget development, let me show a few useful tips with nuget, which we will use.

GeneratePackageOnBuild

In sdk style csproj, we can add <GeneratePackageOnBuild>true</GeneratePackageOnBuild> which can automatically generate the nupkg file for us upon build. This makes it much easier, especially if you come from the times when we had to use nuspec files. Now all references, nuget properties etc can all stay in the csproj. Refer Automatically generate package on build for documentation.

NuGet package folder structure

Microsoft has a convention-based directory structure for the nupkg file. A nupkg file is essentially a zip file and the different folders mean something specific. For example, the one we are going to use the build folder that can have targets and props file which will be triggered during the build of the project that references our nupkg. Note that the props or targets file name in the build folder has to be the same as the package id.

MSBuild props and targets in nupkg

This is what we discussed a bit in the previous point. Any props files in the build folder are imported at the top of the consuming project and any targets are imported at the end of the csproj. The convention is that they have to be named after the package id, else they will be ignored.

IncludeBuildOutput

Another important property which we can include in sdk style csproj is <IncludeBuildOutput>false</IncludeBuildOutput>. This is so that if say the objective of the nupkg is just to ship a targets file (which is what we will do in a bit), it doesn’t need to include any build output. Note that without this property, by default class library projects will include an empty dll in the nupkg.

Demo plan

As presented, in the architecture diagram below, we will have a .net framework project that generates a nupkg. This package is referenced by a second .net framework project. When the second project builds, it automatically triggers the obfuscation of the generated dll.

Code

The full code is on github and I will run through the csproj file here which generates the nuget package to help obfuscate the output dll of the referencing project.

RDD.Obfuscation.AfterBuild.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net48</TargetFramework>
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
    <PackageId>RDD.Obfuscation.AfterBuild</PackageId>
    <IncludeBuildOutput>false</IncludeBuildOutput>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="ConfuserEx.Final" Version="1.0.0" />
  </ItemGroup>
  <ItemGroup>
    <None Include="RDD.Obfuscation.AfterBuild.targets" Pack="True" PackagePath="build" />
  </ItemGroup>
</Project>

The RDD.Obfuscation.AfterBuild.csproj produces the obfuscation helper nuget package. The nupkg is a simple nuget package with a targets file shown below.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>

    <!-- ConfuserToolsPath -->
    <ConfuserToolsPath>$(MSBuildThisFileDirectory)..\..\..\ConfuserEx.Final\1.0.0\tools</ConfuserToolsPath>
    <ConfuserExePath>$(ConfuserToolsPath)\Confuser.CLI.exe</ConfuserExePath>

    <!-- Confuser command -->
    <ConfuserCommand Condition=" '$(OS)' == 'Windows_NT'">"$(ConfuserExePath)"</ConfuserCommand>

    <!-- ProjectFileName -->
    <ProjectFileName>$(ProjectDir)Confuser.crproj</ProjectFileName>

    <!-- Commands -->
    <ConfuseCommand>$(ConfuserCommand) "$(ProjectFileName)"</ConfuseCommand>
  </PropertyGroup>

  <Target Name="Confuser" AfterTargets="AfterBuild">
    <Exec WorkingDirectory="$(ProjectDir)" Command="$(ConfuseCommand)" />
  </Target>
</Project>

The targets file locates Confuser.CLI.exe at nuget packages location and the needed Confuser.crproj at the consuming project’s root. Then it triggers obfuscation after target AfterBuild has finished. Thus ensuring the generated dll gets obfuscated.

MSBuild & dotnet build

To generate the nuget, we can simply run msbuild RDD.Obfuscation.AfterBuild.csproj and it will give us RDD.Obfuscation.AfterBuild.1.0.0.nupkg. We could also pass in parameters such as verion msbuild RDD.Obfuscation.AfterBuild.csproj /P:Version=2.0.0.

Can we do these using dotnet? Yes, we can. Let me demonstrate the relation between msbuild and dotnet by showcasing how nuget restore is implemented.

In 2016, dotnet restore used nuget restore under the hood.

In 2019, this has changed to using msbuild restore.

So, in essence, we are using msbuild under dotnet. But to do msbuild bits, as far as I know, we need to use dotnet msbuild command. Which can be an inconvenience if you have mixed (framework and dotnet) codebase.

Recommendations

Here are a few recommendations if you are using dotnet, msbuild, nuget etc in your development environment.

Migrate to sdk style csproj
Use MSBuild if the codebase is .Net Framework or mix of core/standard and framework, else use dotnet.
Migrate to core/standard where possible
Avoid using multiple tools, msbuild, dotnet and nuget. Very likely you can use just one of those.

Talk video

I presented a talk on this topic at .Net Oxford. Thanks to Dan Clarke, the video of the talk is now published on youtube.

Conclusion

In conclusion, nuget, msbuild, dotnet together give us a powerful combination of tools to make obfuscation or any other similar operations faster, easier & reusable. Any common shared tool, utility, libraries etc can be easily shared using nuget and I would highly recommend we did so instead of direct assembly/project references. And if you have something amazing to share, definitely publish it to nuget.org and share with all.

GitHub Actions - Publishing to nuget.org

Tue, 02 Jun 2020 19:00:00 GMT

GitHub Actions

GitHub provides GitHub Actions to help with automating workflows. We can design CI-CD pipelines and also apply policies on branches using them. The action workflows can be saved to a git repo as a yml file. Along with build, test & deploy, GitHub Actions can help with code reviews, branch management and issue triaging as well.

Basics

The basics of GitHub Actions are covered in Core concepts for GitHub Actions. The overall summary is as follows:

Step: A step is an individual task that can run commands or actions. Actions are the smallest portable building block of a workflow.
Job: A set of steps that execute on the same runner constitute a job.
Workflow: A configurable automated process, consisting of one or more jobs.
Workflow File: The YAML file that defines your workflow configuration with at least one job. This file lives in the root of your GitHub repository in the .github/workflows directory.

Simple action

Let’s have a look at a simple workflow file.

name: ADOSCloneAllRepos CI-CD

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  build:

    runs-on: windows-latest

    steps:
    - uses: actions/checkout@v2
    - name: Setup .NET Core
      uses: actions/setup-dotnet@v1
      with:
        dotnet-version: 3.1.101
    - name: Install dependencies
      run: dotnet restore
    - name: Build
      run: dotnet build --configuration Release --no-restore
    - name: Test
      run: dotnet test --no-restore --verbosity normal
    - name: Upload math result for job 2
      uses: actions/upload-artifact@v1
      with:
        name: ADOSCloneAllRepos
        path: ADOSCloneAllRepos\bin\Release\netcoreapp3.1

This workflow, named as ADOSCloneAllRepos CI-CD, gets triggered on when we push changes to master or create a pull request on the master. This is controlled by the on block,

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

After the on, we need to start the jobs jobs and it needs to specify which kind of agent it needs to run on. Here we specify runs-on: windows-latest. The steps now have a set of uses which are actions from GitHub Marketplace. Some are predefined, such as run which executes a command in a prompt.

The key thing to understand here is, unlike publish build artefacts task in Azure pipelines, GitHub Actions uses actions/upload-artifact@v1. Because theoretically, the workflow is uploading the artifact to GitHub.

So this workflow, builds, tests and publishes a dotnet core artifact.

Automated versioning using GitVersion

A major challenge in CI-CD and package management is versioning. In simple cases we can use GitVersion and we can also use it in GitHub Actions workflow. The steps have to change a bit for this:

    steps:
    - uses: actions/checkout@v2
    - name: Fetch all history for all tags and branches
      run: git fetch --prune --unshallow
    - name: Install GitVersion
      uses: gittools/actions/gitversion/setup@v0.9.2
      with:
          versionSpec: '5.2.x'
    - name: Use GitVersion
      id: gitversion # step id used as reference for output values
      uses: gittools/actions/gitversion/execute@v0.9.2
    - name: Setup .NET Core
      uses: actions/setup-dotnet@v1
      with:
        dotnet-version: 3.1.101
    - name: Install dependencies
      run: dotnet restore
    - name: Build
      run: dotnet build --configuration Release --no-restore -p:Version=${{ steps.gitversion.outputs.semVer }}

After the checkout, we need to fetch all history, hence the run: git fetch --prune --unshallow. Then we install GitVersion 5.2.x, more on this is available at use-actions on the marketplace. Then we run GitVersion,

- name: Use GitVersion
      id: gitversion # step id used as reference for output values
      uses: gittools/actions/gitversion/execute@v0.9.2

Notice that we provide an id for the step so that we can access the output of the step in the build step as ${{ steps.gitversion.outputs.semVer }}.

GitHub action secret management

Secret management is an essential part of any CI-CD pipeline. Especially when it is open source on GitHub. Fortunately, it is very easy to manage our secrets on GitHub Actions. The steps are:

Add secret on repository’s Secrets settings:
Then access secret in workflow file as ${{ secrets.NUGET_ORG_API_KEY }}.

Note that GITHUB_TOKEN is a special predefined secret.

Multijob action

A single workflow can have one or more jobs. For the case we have been working on, let’s have two jobs. One to Build and test and second one to Publish to nuget.org.

jobs:
  build:
    name: Build and test
    runs-on: windows-latest

    steps:
    - name: Setup .NET Core
      uses: actions/setup-dotnet@v1
      with:
        dotnet-version: 3.1.101
    - name: Install dependencies
      run: dotnet restore
    - name: Build
      run: dotnet build --configuration Release --no-restore -p:Version=1.0.0
    - name: Test
      run: dotnet test --no-restore --verbosity normal
    - name: Upload ADOSCloneAllRepos
      uses: actions/upload-artifact@v2
      with:
        name: ADOSCloneAllRepos
        path: ADOSCloneAllRepos\bin\Release\ADOSCloneAllRepos.1.0.0.nupkg

  publish_package:
    name: Publish to nuget.org
    needs: build
    runs-on: windows-latest

    steps:
      - name: Download build artifact
        uses: actions/download-artifact@v1
        with:
          name: ADOSCloneAllRepos
      - name: Setup .NET Core
        uses: actions/setup-dotnet@v1
        with:
          dotnet-version: 3.1.101
      - name: Push package to nuget.org
        run: dotnet nuget push ./ADOSCloneAllRepos/ADOSCloneAllRepos.1.0.0.nupkg -k ${{ secrets.NUGET_ORG_API_KEY }} -s https://api.nuget.org/v3/index.json

The first job, builds, tests and uploads artifact to GitHub. The second publish_package job gets the artifact from the previous job and publishes to nuget.org. Notice the needs: build. That tells the workflow that publish_package has a dependency on build job and hence has to run sequentially. The jobs will run in parallel by default.

Conditional job

We can run the publish_package job conditionally, say only if the merge is to master then only we publish the package by adding if: github.ref == 'refs/heads/master' before steps as below:

  publish_package:
    name: Publish to nuget.org
    needs: build
    runs-on: windows-latest

    if: github.ref == 'refs/heads/master'
    steps:

Final action yml

Let’s put all these together into one final workflow yml. The objective being, we build and test for a push to any branch or PR to master. But we publish only on push to master.

name: ADOSCloneAllRepos CI-CD

on:
  push:
    branches: [ '*' ]
  pull_request:
    branches: [ master ]

jobs:
  build:
    name: Build and test
    runs-on: windows-latest
    # Map a step output to a job output
    outputs:
      semVer: ${{ steps.gitversion.outputs.semVer }}

    steps:
    - uses: actions/checkout@v2
    - name: Fetch all history for all tags and branches
      run: git fetch --prune --unshallow
    - name: Install GitVersion
      uses: gittools/actions/gitversion/setup@v0.9.2
      with:
          versionSpec: '5.2.x'
    - name: Use GitVersion
      id: gitversion # step id used as reference for output values
      uses: gittools/actions/gitversion/execute@v0.9.2
    - name: Setup .NET Core
      uses: actions/setup-dotnet@v1
      with:
        dotnet-version: 3.1.101
    - name: Install dependencies
      run: dotnet restore
    - name: Build
      run: dotnet build --configuration Release --no-restore -p:Version=${{ steps.gitversion.outputs.semVer }}
    - name: Test
      run: dotnet test --no-restore --verbosity normal
    - name: Upload ADOSCloneAllRepos
      uses: actions/upload-artifact@v2
      with:
        name: ADOSCloneAllRepos
        path: ADOSCloneAllRepos\bin\Release\ADOSCloneAllRepos.${{ steps.gitversion.outputs.semVer }}.nupkg

  publish_package:
    name: Publish to nuget.org
    needs: build
    runs-on: windows-latest

    if: github.ref == 'refs/heads/master'
    steps:
      - name: Download build artifact
        uses: actions/download-artifact@v1
        with:
          name: ADOSCloneAllRepos
      - name: Setup .NET Core
        uses: actions/setup-dotnet@v1
        with:
          dotnet-version: 3.1.101
      - name: Push package to nuget.org
        run: dotnet nuget push ./ADOSCloneAllRepos/ADOSCloneAllRepos.${{needs.build.outputs.semVer}}.nupkg -k ${{ secrets.NUGET_ORG_API_KEY }} -s https://api.nuget.org/v3/index.json
      - name: Tag commit
        uses: tvdias/github-tagger@v0.0.1
        with:
          repo-token: "${{ secrets.GITHUB_TOKEN }}"
          tag: "${{needs.build.outputs.semVer}}"

Notice the mapping of job output at build, where it maps GitVersion output to a variable called semVer which can then be accessed at the next job.

build:
    name: Build and test
    runs-on: windows-latest
    # Map a step output to a job output
    outputs:
      semVer: ${{ steps.gitversion.outputs.semVer }}

Conclusion

GitHub Actions are really powerful and make automation easy on GitHub (also on GitHub Enterprise). For full code check azure-devops-clone-all-repos. GitHub Actions documentation is a great starting point as well. Special thanks to Timboski for helping with the project.

Azure pipelines - security & compliance using templates

Tue, 26 May 2020 19:00:00 GMT

Templates in Azure pipelines

Templates are a great way to achieve what we could do using Task groups for builds and releases in classic Azure DevOps pipelines. Using templates we can define reusable content, logic, and parameters.

Template types

There are two types of templates, classified based on their usage.

Type 1: Include/insert template

The include/insert type templates can be used to include content, similar to include directive in many programming languages. Or in the lines of XML include, where the content of one file can be inserted into another.

Type 2: Extend template

The extends templates provide an outer structure of the pipeline and a set of places where the template consumer can make targeted alterations. Think in the lines of inheriting from an abstract class in C#. This kind of extends templates can be used to control what is allowed in a pipeline, the template defines logic that another file must follow.

Enforcing policy - security & compliance

Using the extends templates, we can enforce policies on agent pools or environments.

Usage

A simple usage can be as follows. The start.yml file below is the template which can be extended.

# File: start.yml
parameters:
- name: buildSteps # the name of the parameter is buildSteps
  type: stepList # data type is StepList
  default: [] # default value of buildSteps
stages:
- stage: secure_buildstage
  pool: Hosted VS2017
  jobs:
  - job: secure_buildjob
    steps:
    - script: echo This happens before code 
      displayName: 'Base: Pre-build'
    - script: echo Building
      displayName: 'Base: Build'

    - ${{ each step in parameters.buildSteps }}:
      - ${{ each pair in step }}:
          ${{ if ne(pair.value, 'CmdLine@2') }}:
            ${{ pair.key }}: ${{ pair.value }}
          ${{ if eq(pair.value, 'CmdLine@2') }}:
            '${{ pair.value }}': error

    - script: echo This happens after code
      displayName: 'Base: Signing'

This file takes in the build steps as a parameter and runs a secure_buildstage. As part of that stage, it is doing simple display statements here, but the idea is that it could be some very specific build steps it can perform, viz. build with code signing.

Some steps are generated using template expressions.

${{ if eq(pair.value, 'CmdLine@2') }}:
            '${{ pair.value }}': error

The above expression, for example, says if we try to add a (CmdLine@2 task)[https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/command-line?view=azure-devops&tabs=yaml] then the pipeline will throw an error, essentially fail to build. This could be any other security or compliance requirement we might want to enforce on the pipeline.

The way to use the extends template is from a pipeline YAML file use extends key.

# File: azure-pipelines.yml
trigger:
- master

extends:
  template: start.yml
  parameters:
    buildSteps:  
      - bash: echo Test #Passes
        displayName: succeed
      - bash: echo "Test"
        displayName: succeed
      - task: CmdLine@2
        displayName: Test 3 - Will Fail
        inputs:
          script: echo "Script Test"

Approval and checks

So until now, we have discussed how to enforce a policy after we have extended a template. The important bit is how to enforce that extension. This can be done in two places.

Environment

In Azure pipelines deployment environments we can enable checks that any pipeline deploying to that environment must have extended a specific template.

Agent pool

In Azure pipelines agent pools we can enable checks that any pipeline running on those agents will need to have extended a specific template.

On any of those two settings, environment or agent pool, we can enable template check.

Issues, tips & tricks

Templates are really useful to enforce security and compliance requirements as described above. Although it does create a few issues. Luckily we seem to have solutions or at least workarounds for these.

Complexity

Using templates can mean many seemingly unrelated files are related. The overall pipeline with logic, expressions and parameters can grow very quickly. That adds complexity to the system. The current solution provided by Microsoft are limits set on them. You can find details of the limits in documentation.

Breaking changes

Along with complexity, there is the issue of introducing breaking changes in templates. If a template is used across pipelines and we want to introduce a breaking change, say for a new pipeline, it can still break the older ones. This can be avoided by using Git branch or tag. For example, keep the template in a separate repo as follows:

# template.yml
parameters:
- name: usersteps
  type: stepList
  default: []
steps:
- ${{ each step in parameters.usersteps }}
  - ${{ step }}

And use the template by specifying the type, repository & ref. This locks it down to a specific revision.

# azure-pipelines.yml
resources:
  repositories:
  - repository: templates
    type: git
    name: MyProject/MyTemplates
    ref: tags/v1

extends:
  template: template.yml@templates
  parameters:
    usersteps:
    - script: echo This is my first step
    - script: echo This is my second step

Conclusion

As we have seen till now, templates are a great way of reusing pipeline code. As a bonus, they are now super useful for enforcing security and compliance practices on our software team. I hope this was useful, please do share any thoughts or comments you might have.

Automated webapp deployment - Git to Azure

Tue, 05 May 2020 19:00:00 GMT

GitOps

Have you heard of GitOps yet? I have come across this term on multiple Software Engineering Daily podcasts. And it is amazing how it fits into the idea of robust infrastructure and seamless deployment with infrastructure as code. One of the main essence of GitOps, as far as I understand and also what Atlassian has emphasized is that Git effectively can become one “source of truth”. Meaning for both code and infrastructure. I like this idea and point of this post is about reflecting Git changes on web deployments.

Deploying web app from GitHub to Azure

Pre-requisites

This post assumes that you have at least beginner level knowledge regarding Git, GitHub, Azure, Azure DevOps & Powershell.

Deploying GitHub source to web

The steps are nicely described in Microsoft documentation - Authorize Azure App Service. We need to prepare the repository with right files at the root for the Azure app service to perform the deployment.

This is what the following powershell script does using Azure PowerShell.

[CmdletBinding()]
Param(
    [Parameter()]
    [string]$resourceGroupName = "web-deployment-test-rg",
    [Parameter()]
    [string]$webappname = "webapp-github-deployment-" + $(Get-Random)
)

# Replace the following URL with a public GitHub repo URL
$gitrepo="https://github.com/realrubberduckdev/app-service-web-dotnet-get-started.git"
$location="UK South"

# Connect to Azure subscription
Connect-AzAccount

Get-AzResourceGroup -Name $resourceGroupName -ErrorVariable notPresent -ErrorAction SilentlyContinue
if ($notPresent)
{
    # Create a resource group.
    New-AzResourceGroup -Name $resourceGroupName -Location $location
}

# Create an App Service plan in Free tier.
New-AzAppServicePlan -Name $webappname -Location $location `
-ResourceGroupName $resourceGroupName -Tier Free

# Create a web app.
New-AzWebApp -Name $webappname -Location $location `
-AppServicePlan $webappname -ResourceGroupName $resourceGroupName

# Configure GitHub deployment to the staging slot from your GitHub repo and deploy once.
$PropertiesObject = @{
    repoUrl = "$gitrepo";
    branch = "master";
}
Set-AzResource -PropertyObject $PropertiesObject -ResourceGroupName $resourceGroupName `
-ResourceType Microsoft.Web/sites/SourceControls `
-ResourceName $webappname/web -ApiVersion 2019-08-01 -Force

Essentially the script does the following:

Securely connect to Azure
Create resource group if needed
Create Azure app service
Create Azure web app
Set web app deployment to Github repo master branch

This is just one-time deployment as you may see the master branch is always synced with deployment. Any commits and changes to the web app will be deployed upon merge/push to master. Hence it is a continuous deployment model.

Blue/Green deployment from Github

This is the staging & product slots based deployment. The setup steps are described in Microsoft documentation. Although it is even easier to deploy using our powershell script below.

[CmdletBinding()]
Param(
    [Parameter()]
    [string]$resourceGroupName = "web-deployment-test-rg",
    [Parameter()]
    [string]$webappname = "webapp-github-deployment-with-slots-" + $(Get-Random)
)

# Replace the following URL with a public GitHub repo URL
$gitrepo="https://github.com/realrubberduckdev/app-service-web-dotnet-get-started.git"
$location="UK South"

# Connect to Azure subscription
Connect-AzAccount

Get-AzResourceGroup -Name $resourceGroupName -ErrorVariable notPresent -ErrorAction SilentlyContinue
if ($notPresent)
{
    # Create a resource group.
    New-AzResourceGroup -Name $resourceGroupName -Location $location
}

# Create an App Service plan in Free tier.
New-AzAppServicePlan -Name $webappname -Location $location `
-ResourceGroupName $resourceGroupName -Tier Free

# Create a web app.
New-AzWebApp -Name $webappname -Location $location `
-AppServicePlan $webappname -ResourceGroupName $resourceGroupName

# Upgrade App Service plan to Standard tier (minimum required by deployment slots)
Set-AzAppServicePlan -Name $webappname -ResourceGroupName $resourceGroupName `
-Tier Standard

#Create a deployment slot with the name "staging".
New-AzWebAppSlot -Name $webappname -ResourceGroupName $resourceGroupName `
-Slot staging

# Configure GitHub deployment to the staging slot from your GitHub repo and deploy once.
$PropertiesObject = @{
    repoUrl = "$gitrepo";
    branch = "master";
}
Set-AzResource -PropertyObject $PropertiesObject -ResourceGroupName $resourceGroupName `
-ResourceType Microsoft.Web/sites/slots/sourcecontrols `
-ResourceName $webappname/staging/web -ApiVersion 2019-08-01 -Force

# Swap the verified/warmed up staging slot into production.
Switch-AzWebAppSlot -Name $webappname -ResourceGroupName $resourceGroupName `
-SourceSlotName staging -DestinationSlotName production

Essentially the script does the following:

Securely connect to Azure
Create a resource group if needed
Create Azure app service
Create an Azure web app
Create web deployment slots
Set web app staging slot deployment to Github repo master branch
Finally switch the slots so the current state of master becomes available in the production slot

So now that deployment is done, you can go ahead make a change in master. Did you notice that your change is not reflected in the web pages? Let me explain why. The script configures the staging slot to be synced with master and not the production. That is why the script swaps slots at the end. Meaning we need to do the same when master gets new changes and this is where we can do magic using Azure DevOps.

Azure Pipelines CI-CD

Azure pipelines YAML based pipeline with multiple stages is what we need. The yaml below

triggers on a change to master
does a ‘dummy build & test’ as an example, you should do your proper build, followed by another unit testing phase
because staging slot is already synced with the master branch, we do not need a specific stage for that
the final deploy to production is a gated stage and needs approval.

trigger:
- master

pool:
  vmImage: 'windows-latest'

variables:
  solution: '**/*.sln'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'

stages:
- stage: Build
  jobs:
  - job: Build
    pool:
      vmImage: 'windows-latest'
    steps:
    - script: echo "All build and test pass."
- stage: deploy_to_production
  dependsOn: Build
  jobs:
  - deployment: deploy_to_production
    pool:
      vmImage: 'windows-latest'
    environment: 'deploy_to_production'
    strategy:
      runOnce:
        deploy:
          steps:
            - task: AzureAppServiceManage@0
              displayName: AzureAppServiceManage - Swap slots
              inputs:
                azureSubscription: 'Visual Studio Enterprise (77be59ef-9321-4911-8e68-dde3b63a9e67)'
                Action: 'Swap Slots'
                WebAppName: 'webapp-github-deployment-with-slots-1439770693'
                ResourceGroupName: 'web-deployment-test-rg'
                SourceSlot: 'staging'

This YAML file stays in the git repo and triggers Azure pipelines upon change to master. Team members can test the changes on the staging URL before approving, thereby swapping the slots and deploying changes to production.

Deployment using docker

Another method of automated deployment is using docker settings on Azure app service. For this, we will need a docker image, as shown below. This is the docker file used to deploy this blogging website.

FROM node:12.11.1 AS builder
WORKDIR /dpblog
RUN npm i -g gatsby-cli
COPY package*.json ./
RUN npm install
COPY . .
RUN gatsby build

FROM gatsbyjs/gatsby:latest
COPY --from=builder /dpblog/public/ /pub

This is a multi-stage docker file and the first builder stage is used to build the static gatsby website. Followed by the second stage, where it gets the latest gatsby docker image, which essentially has nginx and hosts the static website by copying it across from the builder.

This will need Azure pipelines again to do the build and deployment.

As the architecture diagram shows, we will need to build a new image with the new static contents. Azure pipelines can build this and push to azure container registry. Then either after validation or approval gates or automated deployment, Azure pipelines can then update the Azure app service docker settings to pick up the new docker image.

Conclusion

Azure, Azure pipelines, Github work seamlessly in these scenarios of web deployment. We can keep all relevant info from code, infrastructure creation to pipeline and more on Git. Making it one “source of truth”.

Github

For full code visit realrubberduckdev/app-service-web-dotnet-get-started.

Convert MediaWiki to Markdown wiki

Sat, 18 Apr 2020 19:00:00 GMT

Introduction

MediaWiki and MD based Wikis

MediaWiki is a popular wiki. It powers Wikipedia as well as used by thousands of other companies and organizations. Although it is quite powerful, extensible, customizable and reliable, the markdown based wikis are equally useful, if not more. The frequent updates and improvements of Azure DevOps wiki is making it a strong contender. The major difference between them will be that ADOS wiki is MD based and MediaWiki is not. This is when we will need to convert MediaWiki articles to MD files.

Converting MediaWiki to Markdown

Export MediaWiki Files to XML

As the first step, we will need to export MediaWiki content to a single XML file. There are multiple ways of doing this.

Option 1: Export content

The simplest one is described at Wikipedia Help:Export.

MediaWiki -> Special Pages -> ‘All Pages’
With help from the filter tool at the top of ‘All Pages’, copy the page names to convert into a text file (one filename per line).
MediaWiki -> Special Pages -> ‘Export’
Paste the list of pages into the Export field.
Check: ‘Include only the current revision, not the full history’
Note: This convert script will only do the latest version, not revisions.
Uncheck: Include Templates
Check: Save as file
Click on the ‘Export’ button.
An XML file will be saved locally.

Let’s call this exported file mediawiki_dump.xml.

Option 2: Dump backup

MediaWiki manual shows how to use the dumpBackup.php script.

Steps:

Log into your mediawiki instance
Find the PHP file …/maintenance/dumpBackup.php and your ../LocalSettings.php file. Then try:

php .../maintenance/dumpBackup.php --conf .../LocalSettings.php --full > mediawiki_dump.xml

Note we can use parameters —include-files —uploads to ensure the exported XML includes all the images and other files. But in our example, this currently doesn’t work.

Convert exported XML to MD

For conversion, we will need to use a PHP script, use Pandoc and we also need Composer for the script environment setup. That is quite a lot of dependencies for us to install and this will be operating system specific, so that adds even more variables to the mix. So what do we do?

You guessed it right, and sure enough, we use docker. The docker file below is from mediawiki-to-markdown/Dockerfile

FROM pandoc/latex
WORKDIR /src
COPY composer.* ./

# Install composer, refer: https://github.com/geshan/docker-php-composer-alpine/blob/master/Dockerfile
RUN apk --update add wget \ 
             curl \
             git \
             php7 \
             php7-curl \
             php7-openssl \
             php7-iconv \
             php7-json \
             php7-mbstring \
             php7-phar \
             php7-xml \
             php7-simplexml \
             php7-dom --repository http://nl.alpinelinux.org/alpine/edge/testing/ && rm /var/cache/apk/*

RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/bin --filename=composer 
# end of install composer

RUN composer install
COPY . .
CMD ["sh"]

As you can see, this docker files gets Pandoc docker image, which already has got PHP from dockerhub. So two of our major dependencies are resolved immediately. Followed by composer, which we install using apk Alpine Linux package management. Finally, after the RUN composer install step, the container will be ready for processing XML files.

The steps to do the conversion now are:

Copy the mediawiki_dump.xml to the same folder as the docker file.
Build the image using the following command:
```
docker build -t MediaWiki2MD .
```
After building the docker image, we run it with a volume mapped for accessing the output files.
```
docker run -v ./output/:/src/output MediaWiki2MD sh -c "php convert.php --filename=mediawiki_dump.xml --output=./output"
```
Note: The convert.php file is at mediawiki-to-markdown/convert.php

This should generate the MD files in the output folder.

Limitations

As discussed earlier, this process still fails to gather images, pdfs or other uploaded documents. That is something I plan to SCP directly from MediaWiki server to the MD files location and write a script to edit the MD files references to the files.

Conclusion

Using docker, we have managed to make the conversion process much easier to manage and reproducible. That takes a big chunk of work out of the process. Although it still needs manual intervention in generating the exported XML and also the final bit of sorting out images and uploaded document links in MD files. If you know of an easier or better way, please do share.

Github

For full code visit realrubberduckdev/mediawiki-to-markdown.

Simple Twitter bot using Azure Logic Apps

Sun, 29 Mar 2020 19:00:00 GMT

Introduction

Azure Logic Apps

Azure Logic Apps is a Microsoft cloud service that helps to schedule, automate, and orchestrate tasks, business processes, and workflows when we need to integrate apps, data, systems, and services across enterprises or organizations.

Twitter bot

Twitter bots are possibly a thing of the past now, yet can be still useful. A twitter bot can automatically perform tweet/retweet/follow etc actions, either blindly or based on certain conditions. There are many ways to design one, using the host of libraries provided by twitter or the Twitter API itself on developer.twitter.com.

Twitter bot on Azure Logic Apps

Requirements

Let’s say we want a twitter bot that retweets every time I tweet something or if it meets the search criteria by containing keywords as follows:

#rubberduckdev
#nuget 
#msbuild 
#azure 
azure 
azure devops 
#rubberduckdev

Development (drag & drop, few clicks or simple steps)

Development of logic apps is as simple as few clicks on the Azure portal. A programming background is useful to understand what is going on and troubleshoot if necessary. But it is not mandatory, well that is how Logic Apps have been marketed.

The steps to take are as follows:

Setup a blank logic app. Follow Microsoft documentation if unsure how it is done.
For logic app trigger, search for twitter and use When a new tweet is posted.
Enter values for When a new tweet is posted dialog box. Notice how Search text is set as a query. This is because it is a search query in the underlying logic app. So, to satisfy our requirements from subtopic above, it should be
```
from:@rubberduckdev OR #nuget OR #msbuild OR #azure OR azure OR azure devops OR #rubberduckdev
```
For logic app action following the trigger, search for twitter and use Retweet.
The Retweet action will need to know the tweet id to be retweeted and this will be obtained from the trigger. Search Tweet id in dynamic content and select it.

That’s it. Now save and our Twitter bot is ready. This bot will retweet any tweet by Twitter handle @rubberduckdev or any tweet that contains any of the keywords listed above. Although by design, this will only trigger once per hour (see limitations subtopic) and hence you may not see retweets straightaway. See the logic app run history for a detailed log.

Limitations

The Twitter logic app connector documentation page provides details on the limitations it comes with. One of the main limiting factors is "Frequency of trigger polls: 1 hour", meaning the logic app will trigger only once an hour.

Conclusion

The azure logic app is useful for a whole lot more than a simple twitter bot. But this is a simple example to get to understand how it works. It comes with a host of connectors, easy to use visual workflow and also many default templates to work with which are described in much detail in Microsoft documentation. Our twitter bot aims to do a simple task of retweeting and so the limitation we discussed above is not a major issue. Other connectors can come with their advantages as well as limitations, and we can also create a user voice if we want specific connectors. With all this in mind, I hope this introduction to logic apps was useful and if you have any comments, thoughts or queries please do comment below or tweet me.

Credits

Cover image from cmarix.

SDK style csproj & .Net framework

Sat, 22 Feb 2020 19:00:00 GMT

Introduction

The sdk style csproj (csharp project style) are becoming the new standard for csproj files. All dotnet core and .netstandard project files are currently using these project styles. Microsoft is also constantly bringing in new improvements for sdk style projects, refer Additions to the csproj format for .NET Core.

SDK style csproj advantages

The sdk style csproj files come with many advantages. The main one being they are being constantly supported and improved by Microsoft. The other technical advantages are as follows.

Less clutter: The csproj no more has to keep a list of files to include, rather can keep a list to exclude. This makes it cleaner and developers do not need to remember to add new files to the csproj. Just placing in the folder of csproj adds it. Thereby less clutter in the csproj file and this reduces noise in git commits too. In addition to this, we do not need any reference to System etc too. Only the project references and package references.
MSBuild /t:restore: With sdk style csproj, we can perform a nuget restore using msbuild target ‘restore’. This is great because we will be dependent on one less tool in our build process.
Generate package on build: We can almost eliminate the need for nuspec files to generate nupkg. I say ‘almost’ because there can be a case when we will still need it, described in Generate nupkg using msbuild. In most cases we can just set <GeneratePackageOnBuild>true</GeneratePackageOnBuild> and generate nupkg from csproj.
Auto reload: We can edit the csproj directly in visual studio without having to unload and reload the csproj and then it can auto-reload as well.

Not for .Net Framework csproj!

Although we have established that sdk style csproj is here for the good, it is not the default project template setting on visual studio. Or to be honest, Microsoft does not ship sdk style csproj for .net framework projects.

There have been many requests in this regard:

And two from me too:

Tweet triggering a discussion on the topic of default project templates.
Visual studio developer community ticket to get an answer from Microsoft.

Unfortunately, as it seems, there is no current plan to implement or ship sdk style projects for the .net framework. The possible explanation can be that, the upcoming .NET 5 is likely to be made a better place to migrate to. Until then we all have a large amount of .net framework code in our repositories.

SDK style .net framework project template

As a simple solution, we now have sdk style project templates visual studio extension SDKStyle.Templates.Net in the market place. Hosted on github.

Conclusion

The SDKStyle.Templates.Net hopefully will be helpful until we get official templates. Until then do help with issue reporting or creating pull requests with any project templates missed there or feedback in general.

Generate nupkg using msbuild

Sat, 25 Jan 2020 11:00:00 GMT

Introduction

Among many other advantages of using sdk style csproj in our solutions, to be able to generate nupkg without a nuspec file is a major one. This is described in Microsoft documentation - Create a NuGet package using MSBuild

I agree that it is a big step towards being able to use msbuild to generate artifacts such as exe, dll and now nupkg. But the issues lies with the fact that the msbuild/csproj always assumes that it is a class library, thereby always adding a dll file to the nupkg.

Issue in detail

So the case we have is, we can generate a specific folder structure in a nupkg using nuspec file. And we hit issues when we try the same using a csproj file, in a bid to remove nuspec file usage.

The case - RubberDuckDev.AfterBuild.nupkg

The overall plan of this nupkg is to just keep a targets file in build folder. So the folder structure is as follows:

RubberDuckDev.AfterBuild.nupkg
|____build
         |____
              RubberDuckDev.AfterBuild.targets

So let us assume that we have nuspec file and we do not use msbuild. In such a case we will follow Run nuget pack to generate the .nupkg file and use a nuspec file RubberDuckDev.AfterBuild.nuspec with the following contents.

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
  <metadata>
    <id>RubberDuckDev.AfterBuild</id>
    <version>1.0.0</version>
    <title>RubberDuckDev targets file</title>
    <authors>RubberDuckDev</authors>
    <owners>RubberDuckDev</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>Targets file from RubberDuckDev.</description>
    <summary>Runs custom targets designed by RubberDuckDev.</summary>
  </metadata>
  <files>
    <file src="build\**\*.*" target="build" />
  </files>
</package>

To generate the nupkg, we just run:

nuget pack RubberDuckDev.AfterBuild.nuspec

Now if we are to do this using a sdk style csproj is we create a csproj file RubberDuckDev.AfterBuild.csproj with nupkg related properties. Which is as follows:

<?xml version="1.0" encoding="utf-8"?>
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
    <PackageId>RubberDuckDev.AfterBuild</PackageId>
    <Title>RubberDuckDev targets file</Title>
    <Authors>RubberDuckDev</Authors>
    <Owners>RubberDuckDev</Owners>
    <Description>Targets file from RubberDuckDev.</Description>
    <Summary>Runs custom targets designed by RubberDuckDev.</Summary>
  </PropertyGroup>
  <ItemGroup>
    <None Include="build\**\*.*" Pack="True" PackagePath="build\" />
  </ItemGroup>
</Project>

To generate the nupkg, we just run msbuild:

msbuild RubberDuckDev.AfterBuild.csproj

The problem is the nupkg generated will have an empty dll preset as csproj is assumed to be a class library by default. So the folder structure is now as follow, which what we did not want.

RubberDuckDev.AfterBuild.nupkg
|____build
         |____
              RubberDuckDev.AfterBuild.targets
|____lib
         |____
              net46
                   |____
                        RubberDuckDev.AfterBuild.dll

This create an unnecessary addition to our nupkg the RubberDuckDev.AfterBuild.dll file.

The solution - nuspec

IncludeBuildOutput - Update on 18th Apr 2020

Recently I found out that nuspec is not necessarily the only solution. We can also disable inclusion of build output in nupkg by adding the following to sdk style csproj.

<IncludeBuildOutput>false</IncludeBuildOutput>

Solution using nuspec

The only way I could get it to work is by using a nuspec and csproj together. That is to use msbuild to generate the required folder structure in the nupkg.

So the csproj finally looks like:

<?xml version="1.0" encoding="utf-8"?>
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
    <NuspecFile>RubberDuckDev.AfterBuild.nuspec</NuspecFile>
  </PropertyGroup>
</Project>

The RubberDuckDev.AfterBuild.nuspec file lives in the same folder as the csproj. Only msbuild-ing this csproj will give us the required nupkg with the correct folder structure and no empty dll file.

Versioning technique

I dislike editing files and patching versions at build time and, in my opinion, we should rather be passing in a property/variable to build process for versioning. We can achieve this using the csproj way of building the nupkg.

For this, the csproj will look as follows.

<?xml version="1.0" encoding="utf-8"?>
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
    <NuspecFile>RubberDuckDev.AfterBuild.nuspec</NuspecFile>
    <Version Condition="'$(Version)' == ''">1.0.0</Version>
    <NuspecProperties>version=$(Version)</NuspecProperties>
  </PropertyGroup>
</Project>

What it is saying is that, pass in a property into nuspec named as ‘Version’. And the default value of ‘Version’ is 1.0.0, unless ofcourse set to some other value.

In order to accept and use the property, the nuspec has to change slightly.

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
  <metadata>
    <id>RubberDuckDev.AfterBuild</id>
    <version>$version$</version>
    <title>RubberDuckDev targets file</title>
    <authors>RubberDuckDev</authors>
    <owners>RubberDuckDev</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>Targets file from RubberDuckDev.</description>
    <summary>Runs custom targets designed by RubberDuckDev.</summary>
  </metadata>
  <files>
    <file src="build\**\*.*" target="build" />
  </files>
</package>

Notice how the version element now uses a property $version$.

With this we can generate RubberDuckDev.AfterBuild.2.0.0.nupkg using the command:

msbuild RubberDuckDev.AfterBuild.csproj /P:Version=2.0.0

And this will also give us the expected folder structure. And no empty dll file.

Advantages of using msbuild

We could achieve the folder structure in the nupkg using the nuget pack technique we discussed earlier. But there are more advantages to using msbuild in my opinion.

One tool to build and generate artifacts. No need to use msbuild for building and nuget for generating nupkg. We can also do a nuget restore using msbuild using command msbuild RubberDuckDev.AfterBuild.csproj /t:Restore. So msbuild can serve multiple purposes.
Easier versioning technique. No need to edit nuspec file during build to generate new versions.
More aligned development, build & test technique using only one tool.

Conclusion

It makes development, build, & test easier if we can have multiple csproj in one visual studio solution and not have to add special cases for nuget pack in our scripts. Although sdk style csproj helps us generate nupkg, it is yet to replace nuspec files. I hope it does.

Is there a better technique to build nupkgs? Do let me know. If you agree/disagree with this post, please comment or tweet about it.

Azure DevOps flaky test identification & reporting

Sun, 08 Dec 2019 11:00:00 GMT

Introduction

We will agree that flaky tests make software development a challenging affair. Such tests add non-determinism and uncertainty to our test suite.

“Non-deterministic tests have two problems, firstly they are useless, secondly they are a virulent infection that can completely ruin your entire test suite.” — Martin Fowler Eradicating Non-Determinism in Tests

Although we find it frustrating, it is inevitable that as a codebase grows and becomes more complicated, we get some flaky tests. Be it because of simple reasons as bad setup/teardown or dependency on third party components to difficult to avoid situations such as infrastructure or concurrency.

Shoutout to Azure advent calendar

This blog post is published as part of Azure advent calendar. I would highly recommend a visit to the advent calendar and check out the other amazing topics published there. All videos on the calendar are at Azure Advent Calendar Youtube channel

Flaky test detection on Azure DevOps

If you have got the same issue as Fry then here’s a video explanation on how to use Azure DevOps to assist in identifying and reporting flaky tests.

Conclusion

If you have any thoughts or comments please do get in touch with me on twitter @rubberduckdev. Or use the disqus plugin below.

Credits

Flaky test banner by Bailey McGinn

AZ900 - Simple Tips for Microsoft Certification Exam

Sat, 19 Oct 2019 11:00:00 GMT

Introduction

As in any other professional field, we know the value of self-improvement. It not only helps us face the competition, but we also get to add value to our field of expertise. This is what inspired me to take the Microsoft certification exam. My journey is far from over as I still have at least two other exams to take. Anyway, this is my first post about my experience of the process and any tips I could share. Please note that as per the NDA we sign before taking the exam, I might not be able to give away a lot. This might be a good thing, so that, you get to experience a lot of it on your own.

Why get a Microsoft Certification?

Goes without saying, Microsoft is already an established brand name in the field of Software Engineering. So being vouched by them for our skills is quite well received by peers, employers, recruiters and more. If my opinion doesn’t sound convincing, Microsoft themselves provide a detailed page on why get certified.

Finding the right certification

Microsoft has revamped its certification website to be Microsoft Learning. It has lots of information including the one we saw earlier ‘Why get certified?’ and more. The simplest way to find what certification suits you is to browse all certifications. There you can filter based on your needs.

Now that you know how to get to the list of available certifications, it is now up to you as to what to choose. Depending on what you are passionate about, or what your current/future role demands are, filter out and select your certification for further details.

Microsoft Certified Azure Fundamentals

The AZ-900 Microsoft Azure Fundamentals exam is an important milestone to get into a Microsoft certification path with Microsoft Azure. This exam measures your ability to understand the following concepts: cloud concepts; core Azure services; security, privacy, compliance, and trust; and Azure pricing and support.

As it stands, the AZ-900 exam is optional if you aim at a certification such as Microsoft Certified: Azure Administrator Associate or Microsoft Certified: Azure Developer Associate. Nevertheless, it is quite the foundation stone upon which you can base your further learning. And if you have not taken any exams with Microsoft, starting with this one will give you more insight into the exam process itself if not just the technical knowledge around Azure fundamentals.

Preparation

The following are the tips on how to prepare for the AZ-900 exam. Based on what I did. As I progress towards more difficult areas of exams, I might adapt these and will do future blog posts about them.

Studying

The initial ‘must-do’, in my opinion, is to start with the self-paced interactive learning path as it would be recommended on the exam page. In the case of AZ-900, it is the Azure Fundamentals Learning Path. This is a gamified learning path, which got me interested for a few days. And I learnt quite a bit, even though I started with a bias that it all might be easy. The fundamentals have some really exciting points which are a must know if you are aiming to be an expert in Azure. Take the concept of Azure Government for instance or why Azure Germany is unique non-government cloud or availability sets vs availability zones. I had no idea about these. In case you do not too, do follow the learning path.

I did search for related topics on Pluralsight but given this is a fundamentals exam, I didn’t find any directly related learning paths on Pluralsight. Will continue checking for future exams and will add to posts then.

Practice Exams

I used the practise exams recommended on the exam page.

I took lots of mock exams. The service provider measureup provides a full history of the practice exams. The pass mark in practice exams is 80% and initially, I scored 67%. But within a few days of those exams, I started scoring around 80%. I aimed to score 100% consistently and then book the real exam. But I did lose patience and when I consistently started scoring 90%+ I booked the exam.

Extras

Listening

The overall idea of the certification exercise is continuous learning and getting vouched for it. So I think that should be the case in other parts of your professional activities as well. For example, if you have got a colleague who knows more about Azure than you, it is a good idea to get help, discuss ideas and listen to them. Listening is very important, followed by filtering and keeping what you believe is important for you.

Local Events

I am lucky to be working at Oxford which means I get to attend events such as .Net Oxford, Azure Oxford or DevOps Oxford. These may not always be doing something directly related to AZ-900 or even Azure. But one of the techniques we all use consciously or subconsciously is taking an educated guess while answering. This is where stimulating our brain comes in which in turn helps when we end up guessing applying certain method, process or logic which we can pick up from like-minded people.

Taking the exam

Having done all the preparation, taking the exam was less stressful than I’d have thought. As far as I know, there are two ways to take the exam.

Exam Center

When you schedule the exam, it leads you through a form to select an exam centre and timing as per the availability. Then on the scheduled time or normally 15mins before your exam, you reach the venue, go through an initial test exam and then take your exam. While scheduling exam I had chosen Pearson VUE, which is what most of us will choose. The other option is Certiport and you should choose that if you are attending a school or are an instructor.

I have also taken Exam 483: Programming in C# in the past at the exam centre in Reading, Berkshire, UK. So this time around, I thought it would be good to try out the second option, BYOD.

Proctored Exam: BYOD

BYOD (bring your own device) proctored exam is new to me and hence I chose this option for my AZ-900 exam.

The way it works is you take the exam on your chosen laptop and someone (or more people) will proctor the exam from a remote location. This means your machine will have to run a test provided by them before the exam starts and camera, microphone etc need to be working and switched on and your location is away from any kind of disruptions.

This is ok if you are taking the exam from an office meeting room where you are certain that nobody will knock or come in. I took the exam from my home office room, which was fine disruption wise, but it felt like letting a stranger(s) into my room. Next exam, I might take it from an office room or exam centre.

Having said so, I can see the benefit of BYOD for people who may not be able to go to the exam centre. It needs further improvements, but I do congratulate MeasureUp on implementing it.

Results

The exam results are generated immediately after the exam and it includes marks scored with a further breakdown of performance.

Having taken the exam, you can get details on your history, generate badges, certificates to share etc at Microsoft Learning Dashboard. Although this is present, I have struggled with the UI in the past and the Pearson VUE (the exam service provider) web site to get exam result details. Hence I always keep a pdf of the exam the moment it gets generated.

The badges are provided by Acclaim for Microsoft. You can view and share the badges from acclaim’s website.

Conclusion

I managed to learn a lot more than Azure Fundamentals while taking the AZ-900 exam. Hoping this post helps others in preparation of a Microsoft exam. If you have any thoughts or comments please do get in touch with me on twitter @rubberduckdev.

DDD14 Reading 2019

Sun, 13 Oct 2019 16:00:00 GMT

Developer! Developer! Developer! Day

The DDD14 held at Reading on 12th Oct 2019 was my first ever experience of Developer Developer Developer community event. It is a fantastic event and I have been missing out by not attending it in the past. The quality of speakers and the content of presentations were amazing, add to it the great organization and location (it was at Microsoft UK HQ at Thames Valley Park). If you are a member of the software community, definitely do not miss out on this great event.

Here are a few of the talks I could attend on the day.

How to steal an election

By Gary Short.

This was a great talk to start with. It comes with a controversial title and goes on to excite and amaze the audience with an explanation of multiple techniques that can be applied and to a large extent is applied to steal an election. By ‘steal an election’, as was explained by Gary, it means “to illegally/immorally influence a result contrary to the current mood of the electorate”.

We may already be aware of many of the techniques used by political parties to try to influence their voters such as Caging: where hostile voters are stripped of their right to vote or discouraged enough to not to vote. Gerrymandering: re-architect district borders to gain a numbers advantage. Tossing: creating situations when votes cast can be discarded as invalid.

Although the extra exciting bit provided was the real world examples used by Gary to show how some parties are currently doing it. He goes to use the audience to participate in something which we would normally wouldn’t agree to do, to prove how influence works.

Overall, both scary and exciting and the full video is available on Youtube.

Air Quality, Azure Functions and Spider Eggs

By Rob Miles.

This talk is actually about IoT devices, use of Microsoft Azure technologies as well as measuring air quality, well just to start with, but was a whole lot more. It was clear how passionate Rob is towards his projects and technology in general and it showed in his presentation when he insisted on trying hobby projects is the best way to learn.

From the air quality measurement side of things, Rob did come with some amazing gadgets and a classy top hat (with air quality sensors on it). To begin with, he demonstrated connecting a device using Message Queue Telemetry Transport with a simplified architecture as below.

Following this, the talk was more about using Azure IoT Hub and Azure Functions for managing MQTT messages and interacting with the devices.

Among many devices, Rob described using LoRaWAN to design cow tracking devices and leaving it our imagination to use it for other creative solutions. All this and more are available at Rob’s github account including all source code.

Internationalising your applications

By Pete Vickers

Microsoft’s Multilingual App Toolkit was the main focal point of this talk. Pete presented how the process of translation is now comparatively easier with the use of the toolkit. The demonstration was on Pete’s Restaurant App. The demonstration included complicated cases of languages which are written from right to left (such as Arabic or Hebrew, although only Arabic was included in the demo), languages with the double-byte character set such as Japanese and also languages that use umlaut or similar notations such as German.

Typescript for the C# developer

By Peter Shaw

Peter presented a great take on Typescript from a C# developer’s perspective. While playing to the dislike we share towards JavaScript due to its anomalistic behaviour, Peter advocated the adoption of Typescript which I wholeheartedly agree with. The funniest bit of the presentation was the clip from Gary Bernhardt’s talk titled Wat, again showing how complicated JavaScript can be. If you have not already seen the video, do take a look at Wat, I promise you that it is hilarious.

While promising not to dive in-depth into the world of Typescript, Peter started with clearing up many misconceptions we might have about Typescript. Followed by the explaining how the goodness of C# which we miss in JavaScript is present in Typescript. Having said so, Typescript has its complications as well, the ‘null vs undefined madness’ for example.

Do check out the slides of the fantastic talk.

.Net Configuration is Easy … Right?

By Steve Collins

Steve starts with a brief overview of the history of configuration in .NET Framework and how things have changed for dotnet core, some for the good while some issues remain.

The best bit of the talk is that it suggests a “SOLID” based approach that makes configuration not only fully testable, but adds enhancements to handle encrypted configuration values. To achieve this, Steve used multiple design patterns and mainly focused on Bridge pattern originally suggested by Gang of Four.

The code for this demo and a link to Steve’s blog post are available at his github.

Conclusion

Overall, the experience at my first ever DDD event was fantastic. I would highly recommend it as it teaches and inspires you, like no other event I know of.