Functional Design in Clojure

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to [email protected], or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "separating data from I/O". We need to test our logic, but the I/O is getting in the way.

We're using Clojure. Everything should be perfect, right?!

I love Hammock Time for figuring out hard problems, but in this case, I think we have a simple problem of testing.

You got to have the right amount of celebration after all those "line crossings" and "goal scorings" and stuff.

We're doing a relatively simple process: we're downloading things and compiling them together into a file. But, it's amazing just how much logic is all throughout this process.

As soon as you make a process, there's always going to be people who want to do it differently!

If experience is any indicator, you always need more information.

One of the reasons why you test is, when you make this kind of logic change, you want to make sure that everything continues to function.

You need to write tests so that when you make future changes, your old self is there sitting right next to you making sure that the old use cases are all covered, so that you only have to think about the new use cases.

With REPLing, you're figuring it out. With tests, you're locking it down and making sure that you have coverage in different situations.

Our biggest obstacle here is that logic and I/O are mixed up together.

Wait! Wait! We want to test our code. We don't want to spend our life writing code. Did you write the mock correctly? How do you write a test for the mock?

I think we need to completely pivot our approach here.

The problem is that we have I/O, logic, I/O, logic, I/O, logic. We have those two things right next to each other. What we should do instead is completely invert our thinking.

Let's gather information and then we can do pure logic on that data. Separate those two things.

We're going to extract from those POJOs. [Groan] I've got to use these terms every now and again or else I'm going to forget them all.

So we do an I/O call, collect information, and create our own internal representation. We just need a few bits of it, so we create a working representation of that.

It's our representation. It's our program's way of looking at the world. Craft the different scenarios in data that represent all the real life situations we found.

One of the problems of using built-ins is: what parts matter?

We're accreting working information into a larger and larger context.

You're setting the table with all the pieces that are defined in your working world and then creating unit tests in terms of those.

The world was like, "Hold my beer!"

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to [email protected], or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "testing around I/O". We start testing our code only to discover we need the whole world running first!

The tracer bullet misfires every now and again.

Now you're going from a tracer bullet to a silver bullet—apparently trying to solve all the problems at once!

The REPL lets you figure out the basics of the process and your own way of thinking about it and modeling it, and the tests let you start handling more and more cases.

Exploration early, testing later.

Are you just supposed to log everything all the time? Always run your code with a profiler attached?

If you look between each I/O step, there is pure connective tissue that holds those things together. We remove the logic and leave just the I/O by itself.

With pure functions, we don't have to worry about provisioning the AWS cluster for the tests to run!

It's really tempting to use the external data as your working data.

What is the data that this application reasons on?

By creating an extractor function, you pull all of the parts that matter into a single place. It returns a map for that entity that you can reason on and schema check.

We've distilled out the sea of information into a drinkable cupful. We've gone from the mountain spring to bottled water.

I guess you could always take all the raw data and shove them off in an Elasticsearch instance for massive debugging later—in some super-sophisticated implementation.

Not how do we accomplish it, but how do we test it?

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to [email protected], or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "handling endless errors". We discover when giving up is the way to get ahead.

We don't need a time zone offset because we know it's in UTC!

In the spirit of building up the language to meet our domain, we can write a pure function!

We want a deterministic way to go from this kind of common information into all the other bits of derived information.

I was really hoping that we would finally be done with the errors and we could just get a highlight clip, but yet again, the world has conspired against us to make our life difficult as a programmer!

Hold on! Hold on! The first thing we should do is run our process again because maybe the error will just go away!

The problem does not go away in this instance. The problem just goes back to being hidden!

The frustrating thing about programming is that code will do exactly what you told it to do.

Clojure is already positive about nothing, that's what we call nil, so why not be positive about bad stuff too?

Now I'm personifying the function as myself!

It's better to reference information by its main identifier as opposed to some derived identifier.

The context is all over the place: some context in memory, some context in the file system, and some intermediate context in the imperative function.

That's the last problem we encounter, right? You never know what the world's going to throw at you!

Why don't we just run it again? Let's just run it again! Maybe it'll be ready now... Maybe now... Maybe it will be ready now...

And then we hit another error which is: the MAM rejects us for making too many requests!

How long do you wait? Should you back off? That adds a lot of complexity to the code at that level.

Because adding idempotency increases the complexity of that part of the solution, we only want to add it where it's necessary.

The error condition is happening right now, so let's write the code to fix it right now.

That's the way automation is at some point in time: a human needs to do it.

You can find yourself in a situation where you're trying to do too many automatic things. Is it really worth doing? You cannot solve every possible permutation once you're interacting with the real world.

It's okay to put up the guardrails, and if I'm outside them, robot me throws up my hands and says, "Human please!"

But what happens when you run it again and the clip is ready? Now your retry logic doesn't get tested.

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to [email protected], or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "building up reliability". We push our software to reach out to the real world and the real world pushes back.

This is very incremental. We are rapidly accreting functionality. We're bringing it together with a very interactive REPL-driven way of getting things done.

There's no reason why we would ever need to modify this code ever again because everything will go smoothly! I'm sure nobody will ever change their mind on functionality and nobody will ever make a mistake in any of the other systems either!

You always have to deal with the uncertainties through time, not just the uncertainties of requirements.

One of the great things about this interactive REPL-driven way is that we are exploring the real world. We're not exploring somebody's documentation or somebody's article or somebody's representation of the real world. We're actually interacting with real systems, and we're looking at real data, and we're figuring out the real situation.

We're not trying to make the ideal version for production. We are trying to get a fully automated solution end to end to understand all the specific situations we have to handle.

That S3 function is built on a tower of abstractions. Some of those abstractions involve the network, and other ones involve other companies. There's a variety of reasons why those might fail—whether for geopolitical or network-based reasons.

The human retry loop is a completely valid solution at this point in time. It's actually a valid solution for a lot of things.

Every time a human has to retry (and the human being is us) we learn every time. We're learning how the systems fail, and that's just as important as the happy path.

Over time, we're going to accrete more and more reliability in the system by handling more and more things.

Deleting the temporary files is a return to known state. It's a pretty harsh return to known state, but it is a return to known state. The initial state of having nothing is a sound state to return to.

I/O is the greatest source of failures when you're automating processes.

Nobody asked your program for permission to turn off the power.

The boss man says, "Make some more! Make some more!"

Let's reduce the recovery time as opposed to trying to avoid the need to recover.

We can convince ourselves that work is needed because we have evidence of failure over time. We're growing functionality on demand as needed. It's very lean.

We're building the right software just in time. Not only are you iterating quickly, so it's not a long time between each change in each rerun, you're also building the right thing every time. It is in the realm of the world, not in the realm of what you think the world is going to do. The actual world is there.

The situation we're in is the real world. It's not something that could happen or maybe happen or "what if" happened.

I/O is the source of pain in our lives, but it's the source of actually making useful software, so it's worth it.

Even if you're in a programming language like Haskell that tries to do proof systems around your logic for handling I/O, it still can't save you from the fact that I/O is going to blow up on you! I/O failures happen.

What happens if you need to retry the retry and then retry that retry? There's only so far you can go with this imperative assembling of the application.

We're letting the reality of our situation dictate where we apply our effort.

We still have learning to do.

Well, that was exceptionally fun!

We're still having fun even though we're encountering errors. Both of those things can happen at the same time!

It's just delightful to see progress. You're always feeling progress! That's a big goal: feel the wind at your back!

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to [email protected], or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "building up a solution". We grow beyond our REPL-driven pieces toward an end-to-end solution.

The learning is complete. Now it's time to get to the programming!

We didn't sign up to be a robot. We signed up to be a programmer.

We want fighting teams. We're not going to have very many highlights if it's the Badgers versus the Doves.

A tracer bullet is a minimalist solution where we try to get something working end to end.

It's interesting that you would say "imperative decomposition", because in this case, we have the parts, so we are doing an imperative composition. We're putting them together.

You get your learning, and you get a little bit of code out of it at the same time. Sure, that code may not be what you want to use in production, but you certainly have more actual code to work with than you did if you just opened up your database explorer and ran SQL statements.

Just because it's a silver bullet doesn't mean all human intervention is no longer needed. A silver bullet has to be fired by someone!

We're growing a function a piece at a time.

So by naming that and giving it a function name, it makes it more readable. It helps document the information.

It's like a mini language here in the let. ... It makes this process—that you're now documenting in code—a little more readable.

You can launch this whole thing using a comment block!

You're exploring your way toward a solution. Even though it's very imperative in this case, you're just continuing to explore your way towards the solution as you build it up.

You're on your second rewrite of the code. You're not writing this code for the very first time. You're writing it for the second time, which means you're going to be picking variable names and function names better than the first time because you understand the concepts. You're not learning the concepts and writing the code at same time.

Exploration was important, but this second step (of making the code again) is valuable, because you're learning what level of abstraction you want.

With command line clients and browsing tools, you still learn a lot about the domain, but you don't learn a lot about how you want to represent it.

It's a coding-first approach, but it's also a coding-light approach. Clojure doesn't make you model the universe in a proof system that you have to try to get right and revise and revise.

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to [email protected], or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "exploration cessation". We realize we're done exploring when all of the pieces fall into place.

That's how I approach problems: you just keep going. Keep moving.

A fiddle is like a random access REPL.

A fiddle is just a file. We can put all the different bits of information—including data—in that one file. There's only one place to go back and look at when we pick up the project the next day.

Well, the MAM actually doesn't store the media. I feel like we were shortchanged!

It's the manager. It doesn't do the heavy lifting. When else have you seen the manager do the heavy lifting?!

Will our troubles never cease?!

There's constructing the request and then there's doing the request.

We're here to explore. We're not here to create bike sheds with bike sheds inside. Yes! This is not the place for abstractions.

We're building up enough language to help us with our exploration. Only write the functions that you need to help you learn more. As soon as you've learned enough, stop writing functions and move on. The point is to keep learning, not to keep abstracting.

You can use the command line, but it's worth doing in the REPL, because you're starting to actually explore the library too.

Utilitarian tends to win in the end. Things that let you get things done quickly. Those solutions are great solutions.

It's good to fiddle because you can play around with it. You can write it the wrong way four times, and get those out of your system before arriving at what you want to use.

The fiddle is here to help you figure it out.

We were doing all this exploring, and we stumbled into doing some actual work!

There's no better way to know you've arrived at the end of your exploring then when all of a sudden, the thing you're trying to do, is now finished!

Over the course of your exploration, you go from exploring to doing actual work. There's no seam between the two activities. Exploring dwindles down as you know more.

Your fiddle is turning into your recipe: a semi-automated way by hand. It's like you've stumbled into a working program.

You're still learning as you go through the process again and again. You're always learning.

You cannot overstate how important it is to experience the variance of the data by hand.

You think, "Oh! I see the pattern!", and then it's example seven that blows up your pattern. Then you think, "Oh, but now I know the pattern!", and then example fifteen blows it up.

You're getting a lot of direct experience with the process. That's going to help you make better abstractions for the application.

You're going to learn it sometime: either now or in the future. It's better to learn it now when you're in exploring mode than later when you're on the hook and your boss is breathing down your neck!

We've sent the asynchronous notification back to the work giver via the Outlook message queue.

It's fun to do the first 10, but after that, you probably want the computer to do all the heavy lifting for you.

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to [email protected], or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "exploring new data and APIs". We peruse APIs to uncover the data hidden beneath.

"This is a situated problem. It's a real-world problem. Like many real-world problems, there are parts we control and parts we don't control. We have to figure out those parts."

"What do we need to know to figure out the information?"

"I like to start with the way people naturally talk about these things. If you start building up your language, it helps to describe things in either the innate input the system must have or the way people talk about it."

"[Honey SQL] will do all the interpolation for you. It's just wonderful! That one feature alone would be enough, but there's plenty more."

"That's a great thing about Hiccup and Honey SQL and other formats that use Clojure data structures to describe things: you get the power of structural editing."

"It's yet another example of building up the vocabulary of the system so that you're able to talk about it at a higher level."

"Practicality is the name of the game here. Imagine you're exploring in a forest and you're trying to figure out what you want to put in your backpack to take along with you. You don't want to take a lot of heavy, complicated tools. You want to only bring along the stuff that is actually useful to you!"

"You only need to build up what is necessary to keep exploring because that's the point. The point is to learn. The point isn't to pre-optimize and make the abstractions."

"My activity is centered around this fiddle file as I'm exploring."

"We have all this cool information, but we're not making database table highlight reels. We're making sports highlight reels in Sportify!"

"How do you make sense of a brand new JSON API that you have never dealt with before?" "By using it."

"This is a good opportunity to mention we have a series called 'Web of Complexity'. The title should give you a sense of what we think of HTTP in general."

"The name web should already be a bad omen! We never use the name 'web' in a positive sense anywhere else."

"Perfect time to use our nonlinear history, aka the fiddle!"

"We've made several really composable ingredients that we're now mixing together as we're learning more about the system."

"Most of my time was spent sifting through the data, not actually making the calls!"

"I'm communicating to myself tomorrow, because I want to forget all this context, go home, do something else, not think about work, and come back to work and pick it all up again."

"One of the benefits of using Clojure to explore is you have a full, rich programming language—one with a full syntax including comments. By doing it all in one fiddle with these comments, you can pick it up in the morning."

"Our lives are nonlinear. We get interrupted. We take a break. We have the audacity to go home and not think about work!"

"It's like a workbench. We're laying out our tools. We're laying out our pieces. Our fiddle file is our workbench and we leave little post-it notes on that workbench to remind us of things."

Here is an example that uses the Cloudflare Streams API. It requires authentication, so we want to factor that out.

First, define some endpoints to work with. Create pure functions for just the part unique to each endpoint.

Then create a function to expand the request with the "common" parts:

For the purposes of illustration, the cloudflare configuration is:

See the full request like so:

Which returns:

Use [babashka.http-client :as http], and call the endpoints like so:

Note, that in a REPL-connected editor, you evaluate each form, so you can see just the check-status-req part, or move out one level and see the full-req part, or move out one more level and actually run it. That lets you iron out the details to make sure you have them right.

Finally, you can make some helpers to use in the REPL or imperative code. The helpers stitch the process together:

They can be called like:

The helpers should do nothing except compose together other parts for convenience. All the real work is in the pure functions.

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to [email protected], or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "using the REPL to explore". We find ourselves in a murky situation, so we go to our REPL-connected editor to shine some light on the details.

"We often want to see highlights more than we want to see the game."

"Like so many things with interns, there is really low supervision, so no one's really worried about how we get it done."

"Having something done is better than not, so the bar is 0."

"It sounds like a natural integration with an asynchronous message queue called 'Outlook'."

"I like to call it 'streaming specification'. I'm not going to tell you everything. I'm coming up with this off the top my head."

"We call them situated problems. They're problems that are set in the real world, so you need to know about all the things in the real world. It's a learning problem."

"You say, 'the things we can't control.' I think of it as, 'the things that are foisted upon us!'"

"We're going to find a way to use Clojure to solve this problem. I bet you didn't see that coming!"

"The first thing I usually do is go for the completely correct and entirely comprehensive documentation that describes all of the different use cases that I might need, and in fact, has a sample code." (Ha! If only!)

"It can be said that programming is debugging an empty file." "The first bug is my application does nothing!"

"You want to get to your first rewrite as fast as possible, so write the messy version first. Then you can learn what you want the next version to be—even if the next version is also messy. The sooner you learn the better."

"I'm starting to build up little pieces to help me explore."

"We're going to figure it out interactively, using the REPL. We're getting our bearings."

"Get in the code right away—hitting an external service as soon as possible—so we can begin to learn, so that we're solving the right problem instead of some problem of our imagination."

"Once it's Clojure data structures, the whole world of Clojure opens up. All the power of mixing and matching and analyzing that data is open to you."

"We're getting into the real thing. We're getting real data, real code. This is going to begin to grow up into something, but for now, we just want to see what's there—begin to explore—get some working code so that we can."

"It's hard to imagine the workflow, because it's not a workflow that you do in other languages. If you haven't done this before, it's hard to just imagine it."

"It's not just REPL first, but REPL-connected editor first, and there's a distinction."

If you are unfamilar with the Clojure command line, check out the Deps and CLI Guide. You can set up global aliases in $HOME/.clojure/deps.edn. For example, this is one Christoph uses:

Run it with:

You can see more examples in Nate's dotfiles and in Sean Corfield's dot-clojure project.

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to [email protected], or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "introducing Sportify!". We tackle a new application, thinking it'll be an easy win—only to discover that our home run was a foul, and the real world is about to strike us out!

Our discussion includes:

Selected quotes:

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to [email protected], or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "thankfulness". We reflect on Clojure, the community, and how much we have to be thankful for.

Our discussion includes:

Selected quotes:

Links: