A joyful programming episode

On the joy of writing a mix of code with PowerShell, YAML, .NET, XML, and XPath. It was a lot of fun.

The task at hand is to write a simple preliminary version of a PowerShell script to create some AWS CloudWatch LogGroups and related LogStreams. The input is an AWS CloudFormation template YAML file. This template file helped before to create the required CloudWatch resources, but now the need is to create the same resources by calls to the AWS CloudWatch Logs API. At execution time, at an AWS EC2 instance, the required PowerShell 7 script will call AWS CloudWatch Logs API by means of the AWS Tools for PowerShell already installed at such EC2 instance, which has attached an AWS IAM Instance Profile with the required permissions.

Curious thing, though, the PowerShell Cmdlets to parse and query the YAML template data were not found as quickly as the Cmdlets to parse and query other common file formats (JSON, XML, CSV, etc.).

PowerShell 7 can load and execute code in .NET Assemblies (given the proper match on targeted .NET CLR). There is, e.g., YamlDotNet package to parse YAML files, but an initial try of it directly from PowerShell was not as straightforward as expected. Nevertheless, that same package could be used from a custom .NET Assembly that would render the YAML content transposed as another serialization format already known to familiar PowerShell Cmdlets. The first use case for such custom .NET Assembly would be like the following test case in an executable functional specification:

	using System.Xml.Linq;
	using Microsoft.VisualStudio.TestTools.UnitTesting;
	[TestClass]
	public partial class ToXMLSpec
	{
	[TestMethod]
	public void from_map1()
	{
	//Arrange
	var map = yaml.deserial("hello: world");
	var expected_xmltext =
	@"<yaml>
	<map>
	<entry>
	<key>hello</key>
	<value>world</value>
	</entry>
	</map>
	</yaml>";
	//Act
	var xml = XDocument.Parse(yaml.AsXml_1dot0(map));
	//Assert
	Assert.AreEqual("yaml", xml.Root.Name);
	Assert.AreEqual(expected_xmltext, $"{xml}");
	}
	}

view raw Spec_case_from_map1.cs hosted with ❤ by GitHub

 

The System Under Test (SUT) is the .NET class named yaml, which encapsulates the access to a YAML parser (deserial method) and to the rendition of the CloudWatch Logs resources data transposed as XML serialization format version 1.0 (AsXml_1dot0 method). This AsXml_1dot0 method can be wrapped by a .NET Console host and invoked from a PowerShell script. The output of that call to the AsXml_1dot0 method can be parsed and queried by Select-Xml Cmdlet.

Next is a general idea for the .NET Console host wrapper for the AsXml_1dot0 method:

	class Host
	{
	static void Main(string[] args) => System.Console.WriteLine(yaml.AsXml_1dot0(yaml.deserial(System.IO.File.ReadAllText(args[0]))));
	}

view raw YAML-to-XML_Host.cs hosted with ❤ by GitHub

 

Before getting into the general layout for the required PowerShell script (Create-CloudWatchLogs.ps1), first let’s see next a basic layout for its invocation and its output:

	PS C:\> .\Create-CloudWatchLogs.ps1 .\CloudWatchLogs.yml
	Creating TopEngine with RetentionInDays = 5
	LogGroup TopEngine : Creating LogStream EngineStats
	LogGroup TopEngine : Creating LogStream InputSession
	LogGroup TopEngine : Creating LogStream Resultset
	Creating MainSource with RetentionInDays = 10
	LogGroup MainSource : Creating LogStream MessageRate
	LogGroup MainSource : Creating LogStream Threshold

view raw YAML-to-XML_output.txt hosted with ❤ by GitHub

 

Also, next is the related input file CloudWatchLogs.yml (an AWS CloudFormation template YAML file) with the data for the resources to be created:

	---
	Resources:
	TopEngine:
	Type: AWS::Logs::LogGroup
	Properties:
	LogGroupName: TopEngine
	RetentionInDays: 5
	EngineStats:
	Type: AWS::Logs::LogStream
	Properties:
	LogStreamName: EngineStats
	LogGroupName:
	Ref: TopEngine
	InputSession:
	Type: AWS::Logs::LogStream
	Properties:
	LogStreamName: InputSession
	LogGroupName:
	Ref: TopEngine
	Resultset:
	Type: AWS::Logs::LogStream
	Properties:
	LogStreamName: Resultset
	LogGroupName:
	Ref: TopEngine
	MainSource:
	Type: AWS::Logs::LogGroup
	Properties:
	LogGroupName: MainSource
	RetentionInDays: 10
	MessageRate:
	Type: AWS::Logs::LogStream
	Properties:
	LogStreamName: MessageRate
	LogGroupName:
	Ref: MainSource
	Threshold:
	Type: AWS::Logs::LogStream
	Properties:
	LogStreamName: Threshold
	LogGroupName:
	Ref: MainSource
	...

view raw YAML-to-XML_CloudWatchLogs.yml hosted with ❤ by GitHub

 

Now, the executable functional specification, so far, includes the case to transpose from a YAML map data type to XML. The YAML file in this case contains only map type instances, but for the case of YAML seq data type, next is an initial test case into the executable functional specification:

	using System.Xml.Linq;
	using Microsoft.VisualStudio.TestTools.UnitTesting;
	public partial class ToXMLSpec
	{
	[TestMethod]
	public void from_list1()
	{
	//Arrange
	var yamltext = @"---
	- a
	- b
	- c
	...";
	var map = yaml.deserial(yamltext);
	var expected_xmltext =
	@"<yaml>
	<list>
	<entry>a</entry>
	<entry>b</entry>
	<entry>c</entry>
	</list>
	</yaml>";
	//Act
	var xml = XDocument.Parse(yaml.AsXml_1dot0(map));
	//Assert
	Assert.AreEqual("yaml", xml.Root.Name);
	Assert.AreEqual(expected_xmltext, $"{xml}");
	}
	}

view raw Spec_case_from_list1.cs hosted with ❤ by GitHub

 

And, next is an initial test case for the case of YAML str data type:

	using System.Xml.Linq;
	using Microsoft.VisualStudio.TestTools.UnitTesting;
	public partial class ToXMLSpec
	{
	[TestMethod]
	public void from_value1()
	{
	//Arrange
	var yamltext = @"---
	a
	...";
	var map = yaml.deserial(yamltext);
	var expected_xmltext =
	@"<yaml>
	<value>a</value>
	</yaml>";
	//Act
	var xml = XDocument.Parse(yaml.AsXml_1dot0(map));
	//Assert
	Assert.AreEqual("yaml", xml.Root.Name);
	Assert.AreEqual(expected_xmltext, $"{xml}");
	}
	}

view raw Spec_case_from_value1.cs hosted with ❤ by GitHub

 

Furthermore, next is a test case for YAML map and YAML seq data types combined:

	using System.Xml.Linq;
	using Microsoft.VisualStudio.TestTools.UnitTesting;
	public partial class ToXMLSpec
	{
	[TestMethod]
	public void from_maplist1()
	{
	//Arrange
	var yamltext = @"---
	a:
	id: 1
	keys:
	- A
	- B
	...";
	var map = yaml.deserial(yamltext);
	var expected_xmltext =
	@"<yaml>
	<map>
	<entry>
	<key>a</key>
	<value>
	<map>
	<entry>
	<key>id</key>
	<value>1</value>
	</entry>
	<entry>
	<key>keys</key>
	<value>
	<list>
	<entry>A</entry>
	<entry>B</entry>
	</list>
	</value>
	</entry>
	</map>
	</value>
	</entry>
	</map>
	</yaml>";
	//Act
	var xml = XDocument.Parse(yaml.AsXml_1dot0(map));
	//Assert
	Assert.AreEqual("yaml", xml.Root.Name);
	Assert.AreEqual(expected_xmltext, $"{xml}");
	}
	}

view raw Spec_case_from_maplist1.cs hosted with ❤ by GitHub

 

Next is the general layout of the required PowerShell script Create-CloudWatchLogs.ps1:

	param
	(
	[Parameter(Mandatory=$true)]
	[System.IO.FileInfo]
	$resources_file
	)
	$resources = C:\temp\host.exe $resources_file | Join-String
	$loggroups = Select-Xml -Content $resources -XPath "/yaml/map/entry/key[child::text()='Resources']/parent::node()/value/map/entry[value/map/entry/key/child::text()='Type' and value/map/entry/value/child::text()='AWS::Logs::LogGroup']" | select -ExpandProperty Node | select -ExpandProperty key
	foreach($loggroup in $loggroups)
	{
	$retention_days = Select-Xml -Content $resources -XPath "/yaml/map/entry/key[child::text()='Resources']/parent::node()/value/map/entry/value/map/entry/value/map/entry[parent::node()/entry/value/child::text()='$loggroup' and key/child::text()='RetentionInDays']" | select -ExpandProperty Node|select -ExpandProperty value
	"`nCreating $loggroup with RetentionInDays = $retention_days"
	#Actual call to AWS here...
	$logstreams = Select-Xml -Content $resources -XPath "/yaml/map/entry/key[child::text()='Resources']/parent::node()/value/map/entry[value/map/entry/value/child::text()='AWS::Logs::LogStream' and value/map/entry/value/map/entry/value/map/entry/value/child::text()='$loggroup']" | select -ExpandProperty Node|select -ExpandProperty key
	foreach($logstream in $logstreams)
	{
	"`tLogGroup $loggroup : Creating $logstream"
	#Actual call to AWS here...
	}
	}

view raw YAML-to-XML_Create-CloudWatchLogs.ps1 hosted with ❤ by GitHub

 

I especially enjoyed the craft of those XPath expressions; it has been a while since my last time with XPath.

Finally, next is the current initial implementation of the yaml class with the parsing and transposition of YAML into XML:

	using System;
	using System.IO;
	using System.Xml.Linq;
	using System.Collections.Generic;
	public static class yaml
	{
	#region Deserialization
	public static object deserial(string yaml)
	{
	using var reader = new StringReader(yaml);
	return deserial(reader);
	}
	public static object deserial(TextReader reader)
	{
	var d = new YamlDotNet.Serialization.Deserializer(); //Provided <PackageReference Include="YamlDotNet" Version="9.1.0" />
	return d.Deserialize(reader);
	}
	#endregion
	#region to XML
	public static string AsXml_1dot0(object graph)
	{
	var xml = transpose(graph);
	var root = new XDocument(new XElement("yaml"));
	root.Root.Add(xml);
	return $"{root}";
	}
	public static object transpose(object x)
	{
	object result = default;
	switch (x)
	{
	case null: result = null; break;
	case string s: result = new XElement("value", s); break;
	case IList<object> _l: result = transpose_list(_l); break;
	case IDictionary<object, object> d: result = transpose(d); break;
	default: throw new Exception($"*** unsupported default ({x},{x?.GetType().FullName}) ***");
	}
	return result;
	}
	public static object transpose(IDictionary<object, object> _m)
	{
	var result = new XElement("map");
	foreach (var pair in _m)
	{
	var entry = new XElement("entry");
	result.Add(entry);
	switch (pair.Key)
	{
	case string s: entry.Add(new XElement("key", s)); break;
	default: throw new Exception($"*** unsupported key default {pair.Key} : {pair.Key.GetType().FullName} ***");
	}
	switch (pair.Value)
	{
	case null: entry.Add(new XElement("value")); break;
	case string s: entry.Add(new XElement("value", s)); break;
	case IList<object> _l: entry.Add(new XElement("value", transpose_list(_l))); break;
	case IDictionary<object, object> _c: entry.Add(new XElement("value", transpose(_c))); break;
	default: throw new Exception($"*** unsupported default ({pair.Value},{pair.Value?.GetType().FullName}) ***");
	}
	}
	return result;
	}
	public static object transpose_list(IList<object> _L)
	{
	var result = new XElement("list");
	foreach (var x in _L)
	{
	switch (x)
	{
	case null: result.Add(new XElement("entry")); break;
	case string s: result.Add(new XElement("entry", s)); break;
	case IList<object> _l: result.Add(new XElement("entry", transpose_list(_l))); break;
	case IDictionary<object, object> _c: result.Add(new XElement("entry", transpose(_c))); break;
	default: throw new Exception($"*** unsupported default ({x},{x?.GetType().FullName}) ***");
	}
	}
	return result;
	}
	#endregion
	}

view raw YAML-to-XML_yaml.cs hosted with ❤ by GitHub