If you are writing Clojure code and find that you need to ensure that you call a cleanup function of some sort, then your first choice should be to use with-open
if possible. However, let’s assume that is not what we are doing. Instead let us assume that invoking a function to create a resource of some sort also returns a function that will cleanup that resource. Something like this:
(defn setup-resource [n]
(println (str "setup " n))
(fn []
(println (str "cleaned up " n))))
To use this we can do the following:
;; bad code
(let [cleanup-f (setup-resource 1)]
(cleanup-f))
;; => setup 1
;; => cleaned up 1
That works fine as long as no exceptions are thrown, but in the following code the cleanup is never called.
;; bad code
(let [cleanup-f (setup-resource 1)]
(/ 1 0)
(cleanup-f))
;; => setup 1
An exception is thrown and our call to cleanup-f
never happens.
To correct this we can use try/finally
.
;; good code
(let [cleanup-f (setup-resource 1)]
(try
(/ 1 0)
(finally
(cleanup-f))))
;; => setup 1
;; => cleaned up 1
So even though an exception is thrown, we still cleanup the resource. Note however that the try must immediately follow the call to setup the resource. If we allow code to creep in between the setup call and the try
then we are vulnerable to not cleaning up the resource.
;; bad code
(let [cleanup-f (setup-resource 1)]
(/ 1 0)
(try
(finally
(cleanup-f))))
;; => setup 1
Even if the additional code is in the let
itself, we are still vulnerable:
;; bad code
(let [cleanup-f (setup-resource 1)
_ (/ 1 0)]
(try
(finally
(cleanup-f))))
;; => setup 1
So the lesson is that the try/finally
must appear immediately after the function that sets up the resource.
There is a related rule regarding dealing with multiple resources. You might, naively, write code like the following.
;; bad code
(let [cleanup-f1 (setup-resource 1)
cleanup-f2 (setup-resource 2)]
(try
(finally
(cleanup-f1)
(cleanup-f2))))
;; => setup 1
;; => setup 2
;; => cleaned up 1
;; => cleaned up 2
While this code apparently works for the happy path, it fails in the face of exceptions.
For example, if the setup function has behavior that causes it to throw when the second resource is created
(defn setup-resource [n]
(when (= n 2)
(throw (RuntimeException.)))
(println (str "setup " n))
(fn []
(println (str "cleaned up " n))))
;; bad code
(let [cleanup-f1 (setup-resource 1)
cleanup-f2 (setup-resource 2)]
(try
(finally
(cleanup-f1)
(cleanup-f2))))
;; => setup 1
The first resource is setup, but never cleaned up.
This same code also is not ideal when one of the cleanup functions throws. For example, if the code behaved like this:
(defn setup-resource [n]
(println (str "setup " n))
(fn []
(when (= n 1)
(throw (RuntimeException.)))
(println (str "cleaned up " n))))
;; bad code
(let [cleanup-f1 (setup-resource 1)
cleanup-f2 (setup-resource 2)]
(try
(finally
(cleanup-f1)
(cleanup-f2))))
;; => setup 1
;; => setup 2
The first cleanup call threw an exception, which in turn caused us not to call the second cleanup function.
This could be addressed by wrapping the cleanup calls in their own try/finally
blocks.
;; good code
(let [cleanup-f1 (setup-resource 1)
cleanup-f2 (setup-resource 2)]
(try
(finally
(try (cleanup-f1)
(finally
(cleanup-f2))))))
;; => setup 1
;; => setup 2
;; => cleaned up 2
Now we see that we do as much cleanup as we can. Even though the call to cleanup-f1
threw an exception we still cleanup the second resource
An alternate pattern is to introduce a new try/finally
for each resource created.
;; change this to fail on resource 2, since it will be cleaned up first in this new pattern
(defn setup-resource [n]
(println (str "setup " n))
(fn []
(when (= n 2)
(throw (RuntimeException.)))
(println (str "cleaned up " n))))
;; good code
(let [cleanup-f1 (setup-resource 1)]
(try
(let [cleanup-f2 (setup-resource 2)]
(try
(finally
(cleanup-f2))))
(finally
(cleanup-f1))))
;; => setup 1
;; => setup 2
;; => cleaned up 1
Again, we did as much cleanup as we could in the face of the exception thrown when cleaning up the second resource.
This final approach matches the first rule that was introduced above, as soon as a resource is created, the next statement needs to be a try/finally
that cleans that resource up.
Finally, consider this pattern in light of an exception in the middle of the code
;; change this back to never throw
(defn setup-resource [n]
(println (str "setup " n))
(fn []
(println (str "cleaned up " n))))
(let [cleanup-f1 (setup-resource 1)]
(try
(let [cleanup-f2 (setup-resource 2)]
(try
(/ 1 0)
(finally
(cleanup-f2))))
(finally
(cleanup-f1))))
;; => setup 1
;; => setup 2
;; => cleaned up 2
;; => cleaned up 1
Even though an exception was thrown in the body of the try, all of the resources are cleaned up.
>The following describes an approach we took on a Clojure system to host our code in a monorepo but allow separate artifacts to be produced for various subsets of the code base. Specifically we used lein as follows:
a root project.clj lists all of the external dependencies used by the monorepo
each sub-project has a project.clj
sub-projects use the lein :parent-project attribute to reference the root project.clj
sub-projects us the lein :only-deps attribute under :parent-project configuration to reference any of the root project dependencies needed. This ensures that the whole mono-repo uses the same version of each external dependency.
sub-projects use the lein :source-paths and :test-paths to reference other sub-projects that it needs from the mono-repo.
For example, the following sub-project references cheshire and clj-time from the root project. In particular, notice this reference only consists of the name, not the version of these external libraries. This sub-project also references two other sub-projects within the monorepo: child-project-B and child-project-C.
(defproject child-project-A "version-is-inherited"
:parent-project {:path "../project.clj"
:inherit [:version :dependencies :repositories :profiles :cljfmt
:plugins :test2junit-output-dir]
:only-deps [cheshire/cheshire
clj-time/clj-time]}
:plugins [[lein-parent "0.2.1"]]
:source-paths ["src" "../child-project-B/src" "../child-project-C/src"]
:test-paths ["test" "../child-project-B/test" "../child-project-C/test"])
I have a hunch that there are other lein features that would allow us to improve the configuration, but this approach has met our needs so far.
>Over the past year I have used Prismatic Schema extensively on a large Clojure project. We really like the aid that if offers in understanding, debugging, and using our code. However, we regularly feel a bit of a let down when an argument to a function is another function. Prismatic Schema doesn’t allow you to say much in these cases beyond: this arg is a function.
To address this we extended Prismatic Schema to allow us to add type annotations to the function arguments in higher-order functions (in addition to several other small extensions). This is done by calling s/fn which expects [output-schema & input-schemas]
Usage examples and details of how it works are here
The type checking for the most part is simply checking that the type of the input function exactly matches the declared type. So the checker largely ignores issues of type hierarchies, covariance, contravariance, etc. As chouser pointed out to me the basic issue this feature raises is that instead of comparing parameter instance objects to declared argument types, this feature requires comparing declared types to declared types. This is one way in which the semantics of this feature are different than “normal” Prismatic schema.
I am torn about the code. On the one hand I am dissatisfied that it is not a full, general solution. On the other hand I can shrug and recognize that for a class of functions and usage patterns it works just fine. So while from a type theoretic perspective it is quite limited, we have started using it and getting benefit from it.
>In the Datomic unsession last night at ClojureWest Stu and Rich hosted a Q&A on Datomic. I appreciated their time and I now feel I have a better grasp of what they are doing with Datomic.
In particular I felt enlightened by Stu’s question: “which provides faster data access, a fast local spinning drive or an SSD attached via a fat network pipe?”
I hadn’t focused on this aspect of their approach. In particular I was not aware that Amazon’s DynamoDB is SSD based.
So the way I understand what Datomic does is that it allows the traditional database server to be distributed across many server machines (i.e. Datomic peers). Writes still go through a single server for consistency, but reads can go against a cluster of machines. This is possible because Amazon’s SSD backed DynamoDB offers network access to storage at speeds that historically required local disks. So as I understand it:
Datomic is at the front of a technology wave that is crashing. Spinning disks are out and SSDs are in.
We can practically think of a distributed SSD cluster on a fat network pipe as the equivalent of a shared hard drive in terms of performance.
This means that we can fire up many servers to read from that “shared & distributed” drive.
Once we can start up many servers we have a distributed database server that offers ACID writes through a single server with failover.
Now that we have a distributed data server we have an abundance of processing on the database server.
With an abundance of database server processing power we can move the application code onto the database servers.
To my way of thinking the description above makes it more clear to me what Datomic is enabling and why it is possible. It seems many people that talk about Datomic get hung up on thinking of Datomic peers as traditional clients. To me it seems that Datomic peers are more like traditional database servers.
I wonder if my description above is correct? I wonder if positioning peers as servers would be an easier “sell”?
>I ported over the first few steps of the in-memory Datomic tutorial to Clojure: https://gist.github.com/2060731
To run this code do something like:
download the datomic zip
follow the datomic README to install the datomic jar to your local maven repo (in the mvn incantation change “datomic.jar” to match the local file name from the datomic zip)
setup a lein project something like (I looked in the datomic pom.xml file to see what version of Clojure was required):
(defproject datomic-tutorial "1.0.0-SNAPSHOT" :dependencies [[org.clojure/clojure "1.4.0-beta3"] [com.datomic/datomic "0.1.2753"]] :dev-dependencies [[swank-clojure "1.4.0-SNAPSHOT"]])
At first I tried to run this on a version of Clojure 1.3 and nothing worked.
I assume that there are Clojure protocols that we can use instead of the Java interfaces but I haven’t tracked them down yet.
>The code from my “Building a DSL in Clojure For Controlling a Lego MindStorm / Arduino Robot" talk at Lambda Lounge on February 2, 2012 is up on github.
The talk had a few points:
The first file to checkout is op_demo.clj. This file introduces the new Clojure type that the DSL is built on, a Clojure “operator”. An operator is similar to a Clojure record, except it presents itself as a List instead of as a Map.
With that as our base, we can now introduce the operators that will make up our language.
Let’s see what we can do with these operators:
This example shows a technique for creating a DSL that nestles tightly down into Clojure.
With respect to teaching kids to program, this approach is a natural fit to the robots. The kids can:
As presented at today’s St. Louis Clojure Cljub meeting.
>I updated the defrecord2 code. It now includes the following:
In the course of doing Clojure development I have made extensive use of records and have extended them in many was. So I was excited to see a proposal from the Clojure Core team for defrecord improvements. It looks like a good start, below are my thoughts and questions about the proposal.
One question that arises is how to understand the difference between the various forms. For example these two forms:
#myns.MyRecord[1 2]
and
(MyRecord. 1 2)
As best I can understand it, the first form would have the validation function applied but the second would not.
Furthermore as a literal form, the first can only include constants. So for instance the following use of a function call would not be valid in the first syntax:
(MyRecord. 1 (+ 1 1))
If you need to write expressions to compute field values and want the validation function to be applied then you use one of the factory functions:
(myns/->MyRecord 1 (+ 1 1))
I assume that the position forms will require all of the fields to be provided? If so, the initialization values will only be relevant to the map forms.
My understanding of the proposal is that the literal record syntax is a general syntax for Java objects. So the object created by
(java.util.Locale. "en" "US")
could be expressed in the literal syntax as
#java.util.Locale["en" "US"]
However, it doesn’t seem to me to be generally possible to know which contructor to use for a given object so I am not sure when this literal syntax will be used for printing (non-record) Java objects.
I realize that the names in the proposals are just placeholders… my preference is to not use “->” (which could easily be confused with the threading macro “->”) in the names and to name the functions with lower-case-dashed versions of the record names instead of CamelCase versions. For my sense of aesthetics this makes the use of record factory functions look like “normal” function calls. Furthermore, I value the ability to specify the name of the factory functions at the time the record is defined.
One of the forms of validation that we have found particularly helpful is to validate the names of the fields passed into the map factory function. If the map contains a key that is not a record field name then an exception is thrown. It is possible to add additional, non field-name keys, with assoc.
In addition I think it is useful to allow a validation function to be defined as part of the defrecord.
I value an option to exclude the namespace from the printed from of the records. Instead of this:
(myns/->MyRecord 1 2)
They would optionally print as:
(->MyRecord 1 2)
This is useful when printing deeply nested trees of records because it trims a potentially long namespace identifier from every object’s output.
Finally, we have found it useful to suppress nil values when printing records with the map factory form. Again this makes the output less verbose.
Beyond the initial creation of records we have found it useful to provide functions to create new record objects from existing objects. For example, the syntax could be:
(def x (myns/map->MyRecord {:a 1}))
(myns/map->MyRecord x {:b 2})
;; -> (myns/->MyRecord 1 2)
By virtue of going through the factory function the validations are applied. As opposed to using assoc directly in which case the validations are not applied.
Related to this is the idea of a universal constructor, e.g. named “new-record”:
(def x (myns/map->MyRecord {:a 1}))
(new-record x {:b 2})
;; -> (myns/->MyRecord 1 2)
This allows a new record object to be created from a record object without knowing the type of the object. We have found this useful for writing generic code to handle record objects.
Finally we have found it useful to define a dissoc function that removes a key from a record object, but produces a record object as the result.
So instead of this default behavior from Clojure:
(class (dissoc (map->MyRecord {:a 1 :b 2}) :b))
;; -> clojure.lang.PersistentArrayMap
We would get:
(class (dissoc2 (map->MyRecord {:a 1 :b 2}) :b))
;; -> myns.MyRecord
We have found it useful to define a record? predicate function that reports whether a given object is a record object:
(record? (map->MyRecord {:a 1 :b 2}))
;; -> true
(record? "hello")
;; -> false
We have extended defrecord to define prewalk and postwalk support and this has proven useful (despite the fact that pre/postwalk are semi-deprecated).
We have extended defrecord to generate multi-method implementations for each record class to participate in a zip-record function that allows zippers to be used to navigate record trees. We have used this feature extensively in our product to manipulate record trees.
We have extended defrecord to support matchure. Specifically all records participate in a multi-method that allows them to be used with a “match-record” of our creation that delegates to matchure if-match . For example:
(match-record [(map->MyRecord {:a 1 :b ?b})
(map->MyRecord {:a 1 :b 2})]
b)
;; -> 2
>
Once you start using Clojure protocols to capture abstractions it is natural to want to define implementations of higher level protocols in terms of the lower level protocols. But, Clojure does not allow protocols to be extended to other protocols. At the Clojure Conj in 2010 Rich Hickey mentioned an approach to this problem in which a protocol is extended to Object as a “catch all”. Then if the protocol is used with an object that satisfies protocol X dynamically extend the class of the object to protocol Y.
For example, consider a low-level protocol for a Dog:
(defprotocol Dog
(bark [_]))
And a higher level protocol for an Animal:
(defprotocol Animal
(speak [_]))
We would like to be able to write something like the following to define how a Dog can participate in the Animal protocol.
(adapt-protocol Dog Animal
(speak [dog]
(bark dog)))
I have written a module, named clojure-adapt, that provides this adapt-protocol capability.
The adapt-protocol call registers the adapter functions in a global map that is keyed by the protocols Animal and Dog. The Animal protocol is extended to the base Object class with implementation functions that consult the global adapter map and dynamically extend the Animal protocol to the classes of objects that satisfy the Dog protocol.
If we have an object that satisfies the Dog protocol, for example, a String:
(extend-protocol Dog String
(bark [s] (str "arf " s)))
Then we can use the Animal functions on the Dog:
(speak "Fido")
=> "arf Fido"
The adapters are only used if the object does not satisfy the protocol. So if a protocol is extended to a class, then the adapters are not used on that class even if objects of the class satisfy the protocol being adapted.
For example, if the Animal protocol and the Dog protocol are extended to the Date class the adapter is not used.
(extend-protocol Dog java.util.Date
(bark [d] (str "dog as of " (.getTime d))))
(extend-protocol Animal java.util.Date
(speak [d] (str "animal as of " (.getTime d))))
(bark (java.util.Date. (long 100)))
=> "dog as of 100"
(speak (java.util.Date. (long 100)))
=> "animal as of 100"
When the Animal function, speak, is called on a Date object the adapter is not used even though the Date satisfies the Dog protocol. This is because the Date class is already participating in the Animal protocol.
The adapt-protocol call is tricky to use during development because once an adapter is “installed” for a class subsequent calls to adapt-protocol do not affect the class. One work-around to this is to refine an adapter by using extend-protocol with a test class. Once the adapter is working properly then register it for use via adapt-protocol.
>