Just Five Mins!

Well, moving single pieces of data from one system to other systems isn’t too complicated. What happens when you need to move batches of data?

This is an old problem and an older scenario for sure - nothing that new here!

Perhaps, just tweaking how you might approach staging data, might help. That is what I am talking about today :)

Well, Tim Abel returns!

My good friend Tim (see the link above for more about him!) kindly joined me to discuss software testing - specifically outside in testing!

This podcast came about after I had listened to Tim’s podcast episode, and this spurred me on to ask a few questions! (well, three actually!)

Since Tim is a contract developer/team lead type of person, he has some wonderful experience working with many teams, and especially larger teams.

For someone like me, who isn’t a real developer these days, I wanted to try to understand where software testing is at these days. I remember doing some TDD (test driven development) and E2E (end to end) testing with some projects in the past, but these days I don’t get the exposure of larger scale software testing principles. It does seem things have moved on quite a bit!

Tim has a great approach about outside in testing from a user perspective. Please check out his full episode (above link) as it is well worth the listen!

Until next time, cheery bye!

Well, YARP - the open-source Microsoft dotnet (or .NET if you are old school!) reverse proxy (Yet Another Reverse Proxy). If you have watched the movie Hot Fuzz or live in the west country UK, then this might steal a laugh:

Hot Fuzz YARP!

So, YARP is used internally by Microsoft and within Azure I believe. It is essentially a web-service you configure to handle routing, authentication, transformations and all of that. The advantage is this is code, your code - you can host it where you need it.

I have done previous episodes on Azure API management/gateways and also Zuplo, but for reference, the links to Zuplo and a new API management service I stumbled on, Treblle are below.

Zuplo API management

Treblle API management

I have used Azure API management gateway a fair bit in the past however, I am a fan of Zuplo right now. I haven’t had chance to play with Treblle as yet so I will update once I have!

Until next time!

P.S: Just Fifteen Mins YouTube will be going live soon :)

Well, another recording, ramble mode enabled (is it ever switched off?).

If 2023 brought AI to the masses, then 2024 seems to be the rise of AI hardware. AI in your pocket (or in this case on your clothes!).

Take a listen to get a quick (ish) overview of two AI devices pushing into our lives :)

The humane AI pin

The new re-branded limitless.ai software and AI pendant

Okay, a “real” ramble for a public holiday (well, here in the UK at least).

dApps, smart contracts, blockchain? what the heck is all of this!

Well, you must have heard of Bitcoin right?

Ethereum (which has been around since approx. 2015) has had this idea of smart contracts where developers can create dApps for various distributed blockchain applications - everything from logistics type stuff to procurement and custom crypto currencies.

Well, Bitcoin has primarily been just a crypto currency - until recently.

BVM (Bitcoin Virtual Machine) is out providing a way to build on the Bitcoin blockchain network (known as L2 or Layer 2) in a similar way to Ethereum.

If you must, take a listen to the ramble in this podcast. I will link to some further resources soon - especially how to build dApps/Smart Contracts, but for now:

Bitcoin BVM

Blockchain/ledge in SQL 2022

dApps

I really need to re-visit the above two older podcasts and drill into the detail a bit - as well as bring them up to date :)

Using .NET/dotnet? need some easy AI? Using at least dotnet 6?

Smart components - from the great mind(s) that brought you Blazor!

Still an experiment at the moment, but a neat way to create drop-in-replacement of say text boxes/text areas and combo/dropdown boxes that use some magic AI.

Use cases are actually quite subtle and useful. Check out the podcast for further info and we have some links below to delve a little deeper if you need to.

Github Repo for smart components

So, more note taking apps!

I am still a steadfast OneNote user and I have tried Notion as well as Microsoft Loop of late.

Obsidian was something a saw a couple of years back but looked really basic. The selling point at the time (and one of the main points still) is your notes are stored locally or wherever you decided to save them i.e. not directly linked to a cloud provider like Microsoft, Google and so on.

Obsidian as a nifty graph built in too - this is where the knowledge management aspect kicks in, and for all its gimmicky premise, it actually very clear and useful IMHO!

Take a listen to learn more.

Obsidian link here

Pricing information (keep in mind it really is FREE for personal use with all features!)

A way better explanation of what is the Markdown Format here :)

For Android, I am using this Folder Sync Pro app - not sponsored BTW! but seems to do the job!I did have an issue with the “two-way” sync TBH however, I just setup two one-way syncs to better manage things anyway! They do have an early release of a desktop app and MAC/Linux etc. coming soon.

Okay, that is that - catch you next time.

… a bit like Michael Jackson’s “filler” song ;)

Well, just a ramble (what’s new?) about a few odd things I have been reading.

Amazon DynamoDB - How many requests per second? !!

In 2021, there was a 66-hour Amazon Prime Day shopping event.

The event generated some staggering stats:

○ Trillions of API calls were made to the database by Amazon applications.

○ The peak load to the database reached 89 million requests per second.

○ The database provided single-digit millisecond performance while maintaining high availability.

All of this was made possible by DynamoDB.

Amazon’s DynamoDB is a NoSQL cloud database service that promises consistent performance at any scale.

History of DynamoDB

In the early years, Amazon realized that letting applications access traditional enterprise databases was an invitation to multiple scalability challenges such as managing connections, dealing with concurrent workloads, and handling schema updates. 

Also, high availability was a critical property for always-online systems. Any downtime negatively impacted the company’s revenue.

There was a pressing need for a highly scalable, available, and durable key-value database for fast-changing data such as a shopping cart. 

Dynamo was a response to this need.

However, there was one drawback of Dynamo. It was a single-tenant system and teams were responsible for managing their own Dynamo installations. In other words, every team that used Dynamo had to become experts on various parts of the database service, creating a barrier to adoption.

At about the same time, Amazon launched SimpleDB which reduced operational burden for the teams by providing a managed and elastic experience. The engineers within Amazon’s development team preferred using SimpleDB even though Dynamo might be more suitable for their use case.

But SimpleDB also had some limitations such as:

* The tables had a small storage capacity of 10 GB.

* Request throughput was low.

* Unpredictable read and write latencies because all table attributes were indexed.

Also, the operational burden wasn’t eliminated. Developers still had to take care of dividing data between multiple tables to meet their application’s storage and throughput requirements.

Therefore, the engineers concluded that a better solution would be to combine the best parts of Dynamo (scalability and predictable high performance) with the best parts of SimpleDB (ease of administration, consistency, and a table-based data model).

This led to the launch of DynamoDB as a public AWS service in 2012. It was a culmination of everything they had learned from building large-scale, non-relational databases for Amazon.

Over the years, DynamoDB has added several features based on customer demand.

The below timeline illustrates this constant progress.

Operational Requirements of DynamoDB

DynamoDB has evolved over the years, much of it in response to Amazon’s experiences building highly scalable and reliable cloud computing services. A key challenge has been adding features without impacting the key operational requirements.

The below diagram shows the six fundamental operational requirements fulfilled by DynamoDB.

Let’s look at each of them in a little more detail.

Fully Managed Cloud Service

A fundamental goal of DynamoDB is to free developers from the burden of running their database system. This includes things like patching software, configuring a distributed database cluster, and taking care of hardware needs.  

The applications can just talk to the DynamoDB API for creating tables. They can read and write data without worrying about where those tables are physically stored or how they’re being managed. 

DynamoDB handles everything for the developer right from resource provisioning to software upgrades, data encryption, taking backups, and even failure recovery. 

Multi-Tenant Architecture

DynamoDB also aims to create cost savings for the customers. 

One way to achieve this is using a multi-tenant architecture where data from different customers is stored on the same physical machines. This ensures better resource utilization and lets Amazon pass on the savings to the customers.

However, you still need to provide workload isolation in a multi-tenant system. 

DynamoDB takes care of it via resource reservations, tight provisioning, and monitoring usage for every customer.

Boundless Scale for Tables

Unlike SimpleDB, there are no predefined limits for how much data can be stored in a DynamoDB table. 

DynamoDB is designed to scale the resources dedicated to a table from several servers to many thousands as needed. A table can grow elastically to meet the demands of the customer without any manual intervention.

Predictable Performance

DynamoDB guarantees consistent performance even when the tables grow from a few megabytes to hundreds of terabytes. 

For example, if your application is running in the same AWS region as its data, you can expect to see average latency in the low single-digit millisecond range. 

DynamoDB handles any level of demand through horizontal scaling by automatically partitioning and repartitioning data as and when needed.

Highly Available

DynamoDB supports high availability by replicating data across multiple data centers or availability zones.

Customers can also create global tables that are geo-replicated across selected regions and provide low latency all across the globe. DynamoDB offers an availability SLA of 99.99% for regular tables and 99.999% for global tables.

Flexible Use Cases

Lastly, DynamoDB has a strong focus on flexibility and doesn’t force developers to follow a particular data model. 

There’s no fixed schema and each data item can contain any number of attributes. Tables use a key-value or document data model where developers can opt for strong or eventual consistency while reading items from the table.

Prompt Engineering is already dead?

Well, who knows! but if you are brave enough to try, Google Collab (to allow you to run heavier workloads in the cloud instead of your own laptop/PC) has a process by Anthropic:

Anthropic is the company behind Claude AI too!

Okay, until next time! As promised, I have an overview of Obsidian Note Taking App coming up soon!

Well, Unicorns aside for a moment, let us delve into the world of API gateways and why you might need one - certainly if you build API’s that is!

I have used Azure API gateway management for quite some years now, since it first appeared on Azure actually! But that’s nothing to brag about since it has changed immensely over recent years.

So why do you need an API gateway? what’s in it for me you might ask. Well, have a listen and found out :)

Thanks to the lovely Scott Hanselman for providing the insight for this JFM.

Work with databases? especially Microsoft SQL Server or Azure SQL? then roll up, roll up for yet more AI - just when you think you can’t eat anymore AI, there is room for just a bit more :)

Optionally (and it is a bit gross in a funny way) it reminds me off Monty Python’s Mr Creosote:

Anyway, back to business!

Even though the name “Azure Data Studio” has “Azure” in it, you can use it as an updated replacement to say SQL Server Management Studio (SSMS). ADS is built on the same foundations as VS Code and therefore can be used cross-platform (yeah, you can use MS-SQL on Linux don’t you know!). In fact, a lot of my local dev work uses MS-SQL 2022 within a Linux based docker container. Saves me having a local instance running and I can switch between versions much easier!

ADS (when you install some “needed” extensions from MS!) is almost on par with SSMS and has a lot of advantages. Because ADS is built on the same foundations as VS Code, you can install extensions. Also, you get really good IntelliSense i.e. completion of commands and so on. You can manage PostgreSQL and MySQL as well as MongoDB and Azure Cosmos DB.

So, no surprise that ADS now has GitHub Copilot :) Yeah, you need to pay for GitHub Copilot, but you do get lots of immediate assistance right there when working with SQL or data structures (as opposed to jumping over to ChatGPT).

Here are some links as noted in the podcast:

Overview of Copilot in ADS

Getting started with ADS