I have a lot of parked projects on GitHub, and a lot of repos which contain examples or snippets that I want to refer back to later. But, this always feels like a bit of a waste, plus with content being scraped to feed the AI machine right now, there’s some content I just don’t want my copyright being ignored on. So, I started shifting some of my projects to Codeberg which provides an equally nice experience, without the additional slop added in recent times. Plus, I get to support an EU based project.

I seem to be dealing with a number of instances lately where I have to get a lot of data into SQL. Most of job involves doing this in various Big Data solutions hosted in the cloud where this happens pretty well at scale, using things like Spark, Azure Data Factory, or Azure Fabric. But recently I’ve needed to do this from local devices to instances of Azure SQL, which is fine but there’s some limitation.

Recently I needed to read a very large file of input data and, for each record, perform an HTTP request to send that data off somewhere else.

There’s a few ways to do this. The first and simplest one is to read all of the lines from the file and then iterate over them, this has one big drawback however. In testing it worked fine, but when you scale this up to a few million records then you start to run into memory limitations! Okay, I need to look at another option.