A joyful programming episode

On the joy of writing a mix of code with PowerShell, YAML, .NET, XML, and XPath. It was a lot of fun.

The task at hand is to write a simple preliminary version of a PowerShell script to create some AWS CloudWatch LogGroups and related LogStreams. The input is an AWS CloudFormation template YAML file. This template file helped before to create the required CloudWatch resources, but now the need is to create the same resources by calls to the AWS CloudWatch Logs API. At execution time, at an AWS EC2 instance, the required PowerShell 7 script will call AWS CloudWatch Logs API by means of the AWS Tools for PowerShell already installed at such EC2 instance, which has attached an AWS IAM Instance Profile with the required permissions.

Curious thing, though, the PowerShell Cmdlets to parse and query the YAML template data were not found as quickly as the Cmdlets to parse and query other common file formats (JSON, XML, CSV, etc.).

PowerShell 7 can load and execute code in .NET Assemblies (given the proper match on targeted .NET CLR). There is, e.g., YamlDotNet package to parse YAML files, but an initial try of it directly from PowerShell was not as straightforward as expected. Nevertheless, that same package could be used from a custom .NET Assembly that would render the YAML content transposed as another serialization format already known to familiar PowerShell Cmdlets. The first use case for such custom .NET Assembly would be like the following test case in an executable functional specification:

	using System.Xml.Linq;
	using Microsoft.VisualStudio.TestTools.UnitTesting;
	[TestClass]
	public partial class ToXMLSpec
	{
	[TestMethod]
	public void from_map1()
	{
	//Arrange
	var map = yaml.deserial("hello: world");
	var expected_xmltext =
	@"<yaml>
	<map>
	<entry>
	<key>hello</key>
	<value>world</value>
	</entry>
	</map>
	</yaml>";
	//Act
	var xml = XDocument.Parse(yaml.AsXml_1dot0(map));
	//Assert
	Assert.AreEqual("yaml", xml.Root.Name);
	Assert.AreEqual(expected_xmltext, $"{xml}");
	}
	}

view raw Spec_case_from_map1.cs hosted with ❤ by GitHub

 

The System Under Test (SUT) is the .NET class named yaml, which encapsulates the access to a YAML parser (deserial method) and to the rendition of the CloudWatch Logs resources data transposed as XML serialization format version 1.0 (AsXml_1dot0 method). This AsXml_1dot0 method can be wrapped by a .NET Console host and invoked from a PowerShell script. The output of that call to the AsXml_1dot0 method can be parsed and queried by Select-Xml Cmdlet.

Next is a general idea for the .NET Console host wrapper for the AsXml_1dot0 method:

	class Host
	{
	static void Main(string[] args) => System.Console.WriteLine(yaml.AsXml_1dot0(yaml.deserial(System.IO.File.ReadAllText(args[0]))));
	}

view raw YAML-to-XML_Host.cs hosted with ❤ by GitHub

 

Before getting into the general layout for the required PowerShell script (Create-CloudWatchLogs.ps1), first let’s see next a basic layout for its invocation and its output:

	PS C:\> .\Create-CloudWatchLogs.ps1 .\CloudWatchLogs.yml
	Creating TopEngine with RetentionInDays = 5
	LogGroup TopEngine : Creating LogStream EngineStats
	LogGroup TopEngine : Creating LogStream InputSession
	LogGroup TopEngine : Creating LogStream Resultset
	Creating MainSource with RetentionInDays = 10
	LogGroup MainSource : Creating LogStream MessageRate
	LogGroup MainSource : Creating LogStream Threshold

view raw YAML-to-XML_output.txt hosted with ❤ by GitHub

 

Also, next is the related input file CloudWatchLogs.yml (an AWS CloudFormation template YAML file) with the data for the resources to be created:

	---
	Resources:
	TopEngine:
	Type: AWS::Logs::LogGroup
	Properties:
	LogGroupName: TopEngine
	RetentionInDays: 5
	EngineStats:
	Type: AWS::Logs::LogStream
	Properties:
	LogStreamName: EngineStats
	LogGroupName:
	Ref: TopEngine
	InputSession:
	Type: AWS::Logs::LogStream
	Properties:
	LogStreamName: InputSession
	LogGroupName:
	Ref: TopEngine
	Resultset:
	Type: AWS::Logs::LogStream
	Properties:
	LogStreamName: Resultset
	LogGroupName:
	Ref: TopEngine
	MainSource:
	Type: AWS::Logs::LogGroup
	Properties:
	LogGroupName: MainSource
	RetentionInDays: 10
	MessageRate:
	Type: AWS::Logs::LogStream
	Properties:
	LogStreamName: MessageRate
	LogGroupName:
	Ref: MainSource
	Threshold:
	Type: AWS::Logs::LogStream
	Properties:
	LogStreamName: Threshold
	LogGroupName:
	Ref: MainSource
	...

view raw YAML-to-XML_CloudWatchLogs.yml hosted with ❤ by GitHub

 

Now, the executable functional specification, so far, includes the case to transpose from a YAML map data type to XML. The YAML file in this case contains only map type instances, but for the case of YAML seq data type, next is an initial test case into the executable functional specification:

	using System.Xml.Linq;
	using Microsoft.VisualStudio.TestTools.UnitTesting;
	public partial class ToXMLSpec
	{
	[TestMethod]
	public void from_list1()
	{
	//Arrange
	var yamltext = @"---
	- a
	- b
	- c
	...";
	var map = yaml.deserial(yamltext);
	var expected_xmltext =
	@"<yaml>
	<list>
	<entry>a</entry>
	<entry>b</entry>
	<entry>c</entry>
	</list>
	</yaml>";
	//Act
	var xml = XDocument.Parse(yaml.AsXml_1dot0(map));
	//Assert
	Assert.AreEqual("yaml", xml.Root.Name);
	Assert.AreEqual(expected_xmltext, $"{xml}");
	}
	}

view raw Spec_case_from_list1.cs hosted with ❤ by GitHub

 

And, next is an initial test case for the case of YAML str data type:

	using System.Xml.Linq;
	using Microsoft.VisualStudio.TestTools.UnitTesting;
	public partial class ToXMLSpec
	{
	[TestMethod]
	public void from_value1()
	{
	//Arrange
	var yamltext = @"---
	a
	...";
	var map = yaml.deserial(yamltext);
	var expected_xmltext =
	@"<yaml>
	<value>a</value>
	</yaml>";
	//Act
	var xml = XDocument.Parse(yaml.AsXml_1dot0(map));
	//Assert
	Assert.AreEqual("yaml", xml.Root.Name);
	Assert.AreEqual(expected_xmltext, $"{xml}");
	}
	}

view raw Spec_case_from_value1.cs hosted with ❤ by GitHub

 

Furthermore, next is a test case for YAML map and YAML seq data types combined:

	using System.Xml.Linq;
	using Microsoft.VisualStudio.TestTools.UnitTesting;
	public partial class ToXMLSpec
	{
	[TestMethod]
	public void from_maplist1()
	{
	//Arrange
	var yamltext = @"---
	a:
	id: 1
	keys:
	- A
	- B
	...";
	var map = yaml.deserial(yamltext);
	var expected_xmltext =
	@"<yaml>
	<map>
	<entry>
	<key>a</key>
	<value>
	<map>
	<entry>
	<key>id</key>
	<value>1</value>
	</entry>
	<entry>
	<key>keys</key>
	<value>
	<list>
	<entry>A</entry>
	<entry>B</entry>
	</list>
	</value>
	</entry>
	</map>
	</value>
	</entry>
	</map>
	</yaml>";
	//Act
	var xml = XDocument.Parse(yaml.AsXml_1dot0(map));
	//Assert
	Assert.AreEqual("yaml", xml.Root.Name);
	Assert.AreEqual(expected_xmltext, $"{xml}");
	}
	}

view raw Spec_case_from_maplist1.cs hosted with ❤ by GitHub

 

Next is the general layout of the required PowerShell script Create-CloudWatchLogs.ps1:

	param
	(
	[Parameter(Mandatory=$true)]
	[System.IO.FileInfo]
	$resources_file
	)
	$resources = C:\temp\host.exe $resources_file | Join-String
	$loggroups = Select-Xml -Content $resources -XPath "/yaml/map/entry/key[child::text()='Resources']/parent::node()/value/map/entry[value/map/entry/key/child::text()='Type' and value/map/entry/value/child::text()='AWS::Logs::LogGroup']" | select -ExpandProperty Node | select -ExpandProperty key
	foreach($loggroup in $loggroups)
	{
	$retention_days = Select-Xml -Content $resources -XPath "/yaml/map/entry/key[child::text()='Resources']/parent::node()/value/map/entry/value/map/entry/value/map/entry[parent::node()/entry/value/child::text()='$loggroup' and key/child::text()='RetentionInDays']" | select -ExpandProperty Node|select -ExpandProperty value
	"`nCreating $loggroup with RetentionInDays = $retention_days"
	#Actual call to AWS here...
	$logstreams = Select-Xml -Content $resources -XPath "/yaml/map/entry/key[child::text()='Resources']/parent::node()/value/map/entry[value/map/entry/value/child::text()='AWS::Logs::LogStream' and value/map/entry/value/map/entry/value/map/entry/value/child::text()='$loggroup']" | select -ExpandProperty Node|select -ExpandProperty key
	foreach($logstream in $logstreams)
	{
	"`tLogGroup $loggroup : Creating $logstream"
	#Actual call to AWS here...
	}
	}

view raw YAML-to-XML_Create-CloudWatchLogs.ps1 hosted with ❤ by GitHub

 

I especially enjoyed the craft of those XPath expressions; it has been a while since my last time with XPath.

Finally, next is the current initial implementation of the yaml class with the parsing and transposition of YAML into XML:

	using System;
	using System.IO;
	using System.Xml.Linq;
	using System.Collections.Generic;
	public static class yaml
	{
	#region Deserialization
	public static object deserial(string yaml)
	{
	using var reader = new StringReader(yaml);
	return deserial(reader);
	}
	public static object deserial(TextReader reader)
	{
	var d = new YamlDotNet.Serialization.Deserializer(); //Provided <PackageReference Include="YamlDotNet" Version="9.1.0" />
	return d.Deserialize(reader);
	}
	#endregion
	#region to XML
	public static string AsXml_1dot0(object graph)
	{
	var xml = transpose(graph);
	var root = new XDocument(new XElement("yaml"));
	root.Root.Add(xml);
	return $"{root}";
	}
	public static object transpose(object x)
	{
	object result = default;
	switch (x)
	{
	case null: result = null; break;
	case string s: result = new XElement("value", s); break;
	case IList<object> _l: result = transpose_list(_l); break;
	case IDictionary<object, object> d: result = transpose(d); break;
	default: throw new Exception($"*** unsupported default ({x},{x?.GetType().FullName}) ***");
	}
	return result;
	}
	public static object transpose(IDictionary<object, object> _m)
	{
	var result = new XElement("map");
	foreach (var pair in _m)
	{
	var entry = new XElement("entry");
	result.Add(entry);
	switch (pair.Key)
	{
	case string s: entry.Add(new XElement("key", s)); break;
	default: throw new Exception($"*** unsupported key default {pair.Key} : {pair.Key.GetType().FullName} ***");
	}
	switch (pair.Value)
	{
	case null: entry.Add(new XElement("value")); break;
	case string s: entry.Add(new XElement("value", s)); break;
	case IList<object> _l: entry.Add(new XElement("value", transpose_list(_l))); break;
	case IDictionary<object, object> _c: entry.Add(new XElement("value", transpose(_c))); break;
	default: throw new Exception($"*** unsupported default ({pair.Value},{pair.Value?.GetType().FullName}) ***");
	}
	}
	return result;
	}
	public static object transpose_list(IList<object> _L)
	{
	var result = new XElement("list");
	foreach (var x in _L)
	{
	switch (x)
	{
	case null: result.Add(new XElement("entry")); break;
	case string s: result.Add(new XElement("entry", s)); break;
	case IList<object> _l: result.Add(new XElement("entry", transpose_list(_l))); break;
	case IDictionary<object, object> _c: result.Add(new XElement("entry", transpose(_c))); break;
	default: throw new Exception($"*** unsupported default ({x},{x?.GetType().FullName}) ***");
	}
	}
	return result;
	}
	#endregion
	}

view raw YAML-to-XML_yaml.cs hosted with ❤ by GitHub

 

Know your design tools — The Performing Half-baked object case

Among other important paradigms in software design, object orientation is a common set of design techniques these days. Hierarchies of classes, derivative class instantiations and virtual calls can be often found on software codebases. Hence, the relevance of the Performing Half-baked object case while designing object-oriented software.

The Performing Half-baked object case occurs where a virtual call is executed onto an unborn or uncreated or uninitialized class instance: the actual behavior can be unexpected, of course.

A couple of examples.

Consider the output of this Windows/Visual C++ code:

#include <iostream>

class AA
{
    public:
    AA()
    {
        std::cout << "AA()\n";
        f();
    }
    virtual void f() { std::cout << "AA.f\n"; }
};

class BB : public AA
{
    public:
    BB()
    {
        std::cout << "BB()\n";
        f();
    }
    virtual void f() { std::cout << "BB.f\n";}
};

void main()
{
	auto x = new BB;
}

For another –different– behavior, consider the output of this .NET/Visual C^# code:

using static System.Console;

class AA
{
    public AA()
    {
        WriteLine("AA()");
        f();
    }
    public virtual void f() => WriteLine("AA.f");
}

class BB : AA
{
    public BB()
    {
        WriteLine("BB()");
        f();
    }
    public override void f() => WriteLine("BB.f");
}

class Program
{
    public static void Main()
    {
        var x = new BB();
    }
}

The Performing Half-baked object is a case that designers of object-oriented languages and runtimes have typically faced. Any designer using those languages and runtimes should be aware of this and other cases during the process of object creation.

Singleton in .NET

If I need a process-level global write-once/read-many state, a global variable, a Singleton, in .NET Runtime, then I keep in mind what I said at the post: Know your design tools — The Singleton case.

It is also relevant to keep in mind:

1. the Visual C^# language specification about Static constructors,

2. the section on Static Constructors of the Visual C^# Programming Guide,

3. the description of what BeforeFieldInit does, and

4. the category (Performance) and the 'When to suppress warnings' section of the CA1810 Rule.

The observations on that old post have been useful for me over the years. Specially while designing stateful software services, where the use of locking, global state, global variables, or Singletons do not hinder throughput. On the other hand, in the context of stateless software services processing a high number of concurrent requests, whose scalability is very important, then –to be clear–, I do not prefer any explicit use of locking, any global state, any global variables or any Singleton.

Here is a demonstrative version of a .NET/Visual C^#-based Singleton derived from the sample code of such old post and a couple of MSTest test cases as evidence of its correctness, included post-conditions for same class instance and for same global value:

#region SUT
/// <summary>
///  https://docs.microsoft.com/en-us/archive/blogs/marcod/know-your-design-tools-the-singleton-case
///  https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/language-specification/classes#static-constructors
///  https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and-structs/static-constructors
///  https://docs.microsoft.com/en-us/dotnet/api/system.reflection.typeattributes?view=net-5.0
///  https://docs.microsoft.com/en-us/dotnet/fundamentals/code-analysis/quality-rules/ca1810
/// </summary>
public sealed class TimepointSingleton
{
    #region Nested static class with access to private outer instance constructor.
    public static class Timepoint
    {
        private static readonly TimepointSingleton instance;

        static Timepoint()
        {
            instance = new TimepointSingleton();
        }

        public static TimepointSingleton Instance { get => instance; }
    }
    #endregion

    #region Outer class for instance-level members of Singleton.
    private DateTime point;

    private TimepointSingleton()
    {
        point = DateTime.Now;
        WriteLine("Timepoint acquired");

        //This simulates resource contention to better
        //show up the multithreading problem when no
        //proper thread synchronization is in place.
        Thread.Sleep(4000);
    }

    public DateTime GlobalTimepoint { get => point; }

    public override string ToString() => $"{GlobalTimepoint:s}";
    #endregion
}
#endregion

Test cases (MSTest) as evidence of the expected and actual behavior:

[TestClass]
public class SingletonSpec
{
    [TestMethod]
    public void two_reads()
    {
        //Arrange
        TimepointSingleton a = null;
        TimepointSingleton b = null;

        //Act
        a = TimepointSingleton.Timepoint.Instance;
        b = TimepointSingleton.Timepoint.Instance;

        //Assert
        Assert.IsNotNull(a);
        Assert.IsTrue(AreTheseTheExactSameObject(a, b));
        Assert.AreEqual($"{a}", $"{b}");

        static bool AreTheseTheExactSameObject(object a, object b) => object.ReferenceEquals(a, b);
    }

    [TestMethod]
    public void concurrent_reads()
    {
        //Arrange
        int read_count = 15;
        using var sync = new ManualResetEvent(false);
        var reads = new ConcurrentBag<string>();
        var instances = new ConcurrentBag<TimepointSingleton>();
        var threads = Enumerable.Range(0, read_count).Aggregate(new List<Thread>(), (whole, next) => { whole.Add(new Thread(operation)); return whole; });

        //Act
        threads.ForEach(t => t.Start());
        sync.Set();
        threads.ForEach(t => t.Join());

        //Assert
        Assert.AreEqual(read_count, reads.Count);
        Assert.IsTrue(reads.All(read => string.CompareOrdinal(read, $"{TimepointSingleton.Timepoint.Instance.GlobalTimepoint:s}") == 0));
        Assert.AreEqual(read_count, instances.Count);
        Assert.IsTrue(instances.All(instance => object.ReferenceEquals(instance, TimepointSingleton.Timepoint.Instance)));

        void operation()
        {
            WriteLine($"[{Thread.CurrentThread.ManagedThreadId}] {DateTime.Now:s} Started");
            sync.WaitOne();
            var global_state = TimepointSingleton.Timepoint.Instance;
            reads.Add($"{global_state}");
            instances.Add(global_state);
            WriteLine($"[{Thread.CurrentThread.ManagedThreadId}] {DateTime.Now:s} Instance value: {global_state} hash: {global_state.GetHashCode()}");
        }
    }
}

Freethinking and free inquiry about software creation

I very much enjoy computer programming. I also enjoy reading authors that talk about it from diverse perspectives. Just as much as practicing philosophy and reading whoever author that talks about philosophy —taken as the basic human impulse to go deeper into what causes doubt and wonder.

So far, I would say that some mechanical and tedious tasks related to computer programming are no longer a major challenge because we now have a lot of good tools helping with those tasks. On the other hand, the intellectual challenge of computer programming remains as a major challenge even today. Of course, now many good lessons are at reach, lessons from thought and experience of the last seven decades. But the intellectual challenge of non-trivial software endeavors remains in the category of major intellectual challenges.

Freethinking and free inquiry about the historical development of software creation is still worthwhile: which theses –or which aspects of them–by original authors are still current? How far I am able to articulate now those main theses of the field?

«At the time I was being groomed to become a very good theoretical physicist, but in 1955 that training was aborted when I decided to become a programmer instead. Not suffering from excessive modesty, I had what I considered a very good reason: I had concluded that programming presented a greater challenge than theoretical physics.»

Under the spell of Leibniz's Dream. Prof.dr Edsger W.Dijkstra.

«I felt that my problems as a programmer were for a large portion beyond the scope of what Polya covered. At first I hesitated to say so aloud, because stressing the exceptional nature of the one's field is usually a sure way of making oneself utterly ridiculous. But after careful consideration I concluded that the intellectual challenge presented by the programming task is, indeed, as unprecedented as the high-speed automatic computer itself.»

"Craftsman or Scientist?" prof.dr.Edsger W.Dijkstra.

Writing good software

Writing good software takes time and a lot of practice, diverse concepts, values –like diligence–, a lot of patience, and other skills. It is kind of similar to master a musical instrument.

It is good to take time everyday to learn and practice: write working code everyday.

Just like music, first you write code everyday to implement usual algorithms (searching, sorting, etc.) by yourself.

Eventually, you gain enough skill to start your own little compositions.

All great systems are made by great people

About great people as the actual source of great systems, I think we should go deeper on what entails to be called “great”, both, people and systems. The act of design, and its related dexterities, is one factor towards such professional greatness. Much of these questions have been thoroughly researched by many authors. I have quoted one of them in the following old post: Good designs come from good designers, good designers come from....

Furthermore, great people are required for great systems on both closely related realms: technical and management. Great teams grouping individual designers of all sorts of artifacts, e.g., test cases or executable software components, should afford great data processing systems; likewise, management should afford great organizational systems as supportive environments for those teams.

Gerald M. Weinberg —and many others— have already pointed out that the history of software development is paved with failed attempts to realize gains in quality and productivity without first creating a supportive environment. To improve bad situations, many managers spend their money on tooling, methodologies, outsourcing, training, application packages, etc., but they rarely spend enough to remove the management that made those situations in the first place or they hardly spend enough to improve them.

As Gerald M. Weinberg also said, we —software developers— have always been a would-be profession, and we will remain a would-be profession until we outgrow our obsession with quick fixes that don’t involve fixing the managers themselves.

Of course, as the saying goes, «you’ll never clear the water until you get the hogs out of the creek». But, let’s be crystal clear: such hogs are not bad people, but bad ideas. In other words, many underlying theories of software development management are obsolete; including many theories of project management.

This is already known since decades ago. I just recall an old post about it: The Underlying Theory of Project Management is Obsolete.

Back to basics: Demanding reading

The following article came to my attention: The Satir Change Model.

An article that includes references to the works of Virginia Satir and Gerald M. Weinberg is a promising reading. Of course, the direct reading of the seminal works by those authors –among other luminaries– would be the best reading. Though the closer to the seminal authors, the more demanding the reading would be.

That kind of demanding reading is what I am talking about when I say that we —our profession— need to go back to the basics.