Dynamic Database Connections in Clojure

Someone recently posted this to the clojure mailing list:
I found passing around the database connection to each function that uses it very error prone when you are using transactions as passing the wrong one could mean a query runs outside the transaction when in the source code it is inside the with-db-transaction function.
The general consensus boiled down to “Don’t do that. Go back to passing the database connection through your request handler chain as a function parameter instead.”
I’ll go ahead and make the point that it’s error-prone for different reasons.
Pretty much by definition, that database connection is a system boundary. It’s all about something that’s *way* more complex than random global state changes inside your program. This is a thing that interacts with the outside world, with all the nastiness that implies.
Everything that everyone else has already written about this approach is true, but I don’t think they’ve gone far enough.
Even if you pass that database connection around as a parameter everywhere, you’re talking about throwing away a huge part of the benefit of using a functional language.
Isolate your side-effects.
Think of a castle. You have a moat surrounding it, and a few gates that you use to allow your peasants to enter/exit. This particular gate opens up to a swamp full of alligators.
Your approach amounts to letting the gators wander around loose.
Passing the connection around to each function in the call chain is like tying a ribbon around the gator’s neck and hoping you can use that as a leash.
You can use either approach to great effect. If you’re really, really good. And so is everyone else on your team (you did mention a 200 KLOC project).
One of the main benefits to functional programming is that admitting you aren’t really, really good is incredibly liberating. I don’t have the time/energy to dedicate to trying to maintain this sort of code. (Yes, I spent lots of time recently thinking about how java was designed for very average programmers, but it really takes much better programmers than a functional language to actually write correct programs). Even if I were that good…I’d rather be focused on the problems that make my customers happy.
I’m going to appeal to authority here for the right answer: http://prog21.dadgum.com/23.html (in my defense, it’s a great blog). Have your web response handler (which is another system boundary…this one is next to an active volcano populated by fire-breathing dragons) build up a list of all the nasty side-effects that will eventually have to happen.
Don’t just isolate your side-effects. Quarantine those suckers as if each and every one means you’re dealing with the most diabolical hacker you can imagine.

Datomic Using Postgres Back-End

This started out as a post to the datomic mailing list, asking for help.

I try to ask not-stupid questions, so I kept picking at the obvious “what happens if you do x?” style questions until I wound up arriving at an actual answer.

Background

This may not even be a supported configuration, but it seemed like it might be worth mentioning, if only for its “that seems odd” factor. If nothing else, maybe it’ll help anyone else who runs into this and tries to google for it.

I recently set up datomic pro starter using postgres as the back-end on ubuntu 14.04, following the instructions at http://docs.datomic.com/storage.html. I don’t remember having any issues at all.

A couple of weeks ago, I set up a vagrant box based on the latest 14.04, and have had some issues getting datomic working with postgres 9.4.

First Strangeness

This time around, I couldn’t just run datomic’s bin/sql scripts specifying -U postgres. I had to actually sudo over to that user first.

Transactor Failure

After that was configured, the initial output from starting the transactor (I haven’t changed anything in the default sql config file except adding my license key) looked normal at first:

Launching with Java options -server -Xms1g -Xmx1g -XX:+UseG1GC -XX:MaxGCPauseMillis=50
Starting datomic:sql://<DB-NAME>?jdbc:postgresql://localhost:5432/datomic?user=datomic&password=datomic, you may need to change the user and password parameters to work with your jdbc driver ...
System started datomic:sql://<DB-NAME>?jdbc:postgresql://localhost:5432/datomic?user=datomic&password=datomic, you may need to change the user and password parameters to work with your jdbc driver

After about a minute, it exited with:

Critical failure, cannot continue: Lifecycle thread failed
java.util.concurrent.ExecutionException: org.postgresql.util.PSQLException: ERROR: relation "datomic_kvs" does not exist
  Position: 31
       at java.util.concurrent.FutureTask.report(FutureTask.java:122)
       at java.util.concurrent.FutureTask.get(FutureTask.java:188)
       at clojure.core$deref_future.invoke(core.clj:2180)
       at clojure.core$future_call$reify__6320.deref(core.clj:6420)
       at clojure.core$deref.invoke(core.clj:2200)
       at datomic.lifecycle_ext$standby_loop.invoke(lifecycle_ext.clj:28)
... (my apologies for not including the entire stack trace. I haven't managed to get copy/paste working out of this VM yet)

I’m not all that familiar with postgres. It took me a while to figure out that the trick to connecting to the datomic database as the datomic role via the command line is to run:

psql datomic -h 127.0.0.1 -d datomic

This is really the first time I’ve used the command-line client. The part that caught my eye was that, as the postgres user, running \dt showed something like

                    List of relations
  Schema |      Name       |  Type  |  Owner
-------------+------------------+---------+----------
   public    | datomic_kvs | table    | postgres

Running that same command as the datomic user printed

"No relations found."

My current guess about what went wrong is that I had a typo when I ran the postgres-table.sql script, and it created the datomic_kvs table in the postgres database.

I ran that “create table …” command manually, and the transactor seems to be running without a hitch.

I don’t know whether there might be any impact from using the 9.3 drivers against a 9.4 install. It seems dubious, but I just don’t know enough about postgres to even start trying to hazard a guess on this one.

Client Side

After I wrote that and got a good night’s sleep, I tackled the interesting part: the client.

Trying to create a new database results in:

HornetQConnectionTimedOutException HQ119013: Timed out waiting to receive cluster topology. Group:nell org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory (ServerLocatorImpl.java:946)

I switched ports in the connection string, just for grins to see what would happen. That changed the problem to an EOFException.

So I tried out the basic commands from the “getting started guide” using the groovysh script that comes with datomic. That part worked fine.

When I update the connection string to the database that I created that way and just try to connect through a clojure REPL, the error switches to

CompilerException clojure.lang.lang.ExceptionInfo: Error communicating with HOST localhost on PORT 4334 {:alt-host nil, :peer-version 2, :password "...", :username "...", :port 4334, :host "localhost", :version "0.9.5153", :timestamp 1427687038138, :encrypt-channel true}, compiling:(form-init1256469964991696942.clj:1:17)

It looks like that came from nrepl middleware. It’s

Caused by:
ExceptionInfo: Error communicating with HOST localhost on PORT 4334 {:alt-host nil, :peer-version 2, :password "...", :username "...", :port 4334, :host "localhost", :version "0.9.5153", :timestamp 1427687038138, :encrypt-channel true}
  clojure.core/ex-info (core.clj:4577)
  datomic.connector/endpoint-error (connector.clj:52)
  datomic.connector/create-hornet-factory (connector.clj:148)
  datomic.connector/create-transactor-hornet-connector (connector.clj:327)
  datomic.connector/create-transactor-hornet-connector (connector.clj:325)
  datomic.peer.Connection/fn--8924 (peer.clj:235)
  datomic.peer.Connection (peer.clj:,221)
  datomic.peer/create-connection/reconnect-fn--8985 (peer.clj:488)
  datomic.peer/create-connection (peer.clj:498)
  datomic.peer/get-connection/fn--9039/fn--9041 (peer.clj:665)
  datomic.peer/get-connection/fn--9039 (peer.clj:662)
  datomic.peer/get-connection (peer.clj:656)
Caused by:
HornetQConnectionTimedOutException HQ19013: Timed out waiting to receive cluster topology. Group:null
  org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory (ServerLocatorImpl.java:946)
  datomic.hornet/create-session-factory (hornet.clj:220)
  clojure.core/apply (core.clj:628)
  datomic.connector/try-hornet-connect (connector.clj:108)
  datomic.connector/create-hornet-factory (connector.clj:143)

There are a handful of google results about that final issue, but they all seem to ultimately wind up with the author answering “Oh, I see what I did wrong.

Which leaves me hopeful, but isn’t very helpful.

I headed back to the project roots. I figured it would be simpler to just create a new project, verify that it can connect with nothing but the bare bones, and then add in one piece at a time to see where it starts failing.

The very first difference that stood out was that, without doing anything special, datomic pro’s hornetq-server “2.3.17.Final” overrides immutant’s “2.4.5.Final”. And then that leads to datomic’s hornetq-journal “2.3.17.Final” overriding immutant’s hornetq-journal “2.4.5.Final”. And there are a slew of others along the same lines.

My inclination here is to update immutant, but they’re already using newer libraries.

Instead, I trimmed back that project definition so the datomic-pro dependency got rid of its hornetq-server exclusion. At first glance, that seems to have done the trick.

My initial guess is that the root cause is that hornetq 2.4.x won’t communicate with 2.3.x, but that seems like a really bad guess. I know even less about hornetq than I do about postgres, so that may not even be in the right ballpark. If I had more time to devote to this sort of thing, I’d try to track this down to get a definite answer. As things stand, I’m happy to have an answer that’s good enough to let me move on.

Clojure’s EDN Tagged Literals

We had an issue earlier where the EDN reader was breaking because it didn’t recognize the #bin tag.

The root problem there was that we were sending identity frames across the wire when we shouldn’t have been.

But it’s worth knowing about how to treat the symptom for cases where we want to send things that aren’t legal natively–that’s one of the main points to using an extensible format.

I think clojure introduced customizable “tagged literals” in 1.4. Pretty much all the google results about this talk about the full-blown clojure version. They’re all about setting up a data_readers.clj and using it to describe what should happen when the reader encounters a tag it doesn’t recognize.

We’re just serializing everything by calling pr-str on it. This mostly creates strings that the clojure reader can automatically parse. For things that it can’t, there’s a multi-method (I think it’s print-dup) that you can override.

The clojure reader gracefully handles things that the EDN reader can’t, by default. The EDN reader also ignores data_readers.clj. It’s very specifically designed for deserializing input from untrusted sources.

The danger (and another of the big reasons for using EDN–before it was introduced, the default lisp approach was just to trust the incoming data and use lisp as the serialization format) is that you can have a tag cause the reader to execute any arbitrary code that strikes your fancy.

That’s really a scary amount of power (which can be used for a lot of Good).

Anyway. That’s why the code sample from this morning was complaining about the #bin tag. EDN, by default, doesn’t recognize the format for raw byte arrays. That’s one that we probably shouldn’t ever mess with.

Clojure Baseline

I’ve been trying to wrap my head around clojure for about 2 years now.

I *get* the basic syntax and functional programming ideas. I’m still tying my head around the jvm.

We have a very basic web app front end that’s serving up a bunch of javascript that calls back into another web app, running in the same container.

Pretty much every example I’ve been able to find assumes that you’re using something like Tomcat, with a ton of XML and lots of JSP with annotations.

That’s the sort of thing that drove me away from clojure in the first place.

Well, that and the whole “immutable data structure” thing.

I’ve spend at least 3 hours today trying to deploy immutant servlets into a wildfly container running on a public cloud.

Actually, I’ve spent almost 3 days on this now. Day 1 involved getting one servlet to respond to basic GET/PUT requests. Day 2 was spent translating our designer’s code (no offense to Rick…this isn’t his fault by any means! I should have found time to get a better grasp on details about HTTP headers). And, so far, Day 3 is devoting itself back to java basics that I’ve been thrilled to gloss over.

Like the really basic “java -cp my.jar clojure.main”

You really need to run something like `lein uberjar` to generate that my.jar in the first place.

And you should probably spend some time studying java and the way the CLASSPATH works.

But that’s time well spent. At least, I really hope it will be. Since I’m still trying to figure out how to serve up public/index.html.

I really seems like this should be obvious.

Thanks, Java!

Classlojure Basics

I’m trying to hack around lwjgl’s basic limitations: it *really* wants to be a singleton. I’m hoping I’ll be able to use libgdx as a wrapper on top of it, but they seem to make matters worse because Android seems to be even fussier about the idea that you have 1 Application and 1 Window.

I pretty desperately want multiple windows.

I think that’s on Lwjgl’s roadmap, though I have seen posts from the developers that boil down to “If you want to do something like that, you aren’t building games. Go use something like jogl instead.”

I’m really hoping I’ll be able to work around this by using multiple class loaders to fool the library into believing that I’m running each instance as a singleton.

Classlojure seems like the perfect tool to make that happen, but the documentation leaves a bit to be desired.

The basic idea is simple enough: build a ClassLoader by calling something like

(def cl (classlojure/classlojure "file://${home}/.m2/repository/org/clojure/clojure/1.6.0/clojure-1.6.0.jar"))

and then call

(classlojure/eval-in cl ${something} ${args})

to actually use it.

The docs are a little fuzzy on what that ‘something should be. It really sounded like I wanted to make it a function call. It still seems like that’s probably what I ultimately want, but the unit tests don’t bother.

Based on the unit tests, I almost have a piece working. The ${something} expands out to

'(do (require 'my-project.blah) (my-project.blah/foo))

which has the same problem I ran across when I was providing a function. It can’t find the namespace. (It keeps throwing a FileNotFoundException).

Which is pretty frustrating. Since the file is in the same directory as the one that’s doing the require. (System/getProperty “java.class.path”) tells me that this namespace is chock-full of everything I’m used to dealing with.

The answer to this dilemma is also hidden in the unit tests. Specifically, the require-srcfile test. Though it’s subtle enough that it took me a while to notice (I blame exhaustion brought on by the newborn).

The classlojure function really  takes an ISeq of URLs that are important. I haven’t dug into what the code’s actually doing yet. But the practical upshot is that first “basic idea” form needs to look more like:

(def cl (classlojure/classlojure "file:src/" "file:/path/to/clojure/jar"))

I’m still getting errors about things that my source file is trying to require that should be on the classpath. So I’m guessing that I need to manually add in the rest of the pieces that I know I need, trimming out everything that I’m using as a luxury. This seems like the sort of thing that I’m used to having leiningen handling for me. Messing around with the classpath was one of the main reasons I avoided clojure for so many years.

I’m more than a little curious about whether Chas Emerick’s pomegranate library is the solution to this particular itch.

Either way. I think I see the light at the end of this particular tunnel. Even though it’s still definitely a work in progress.

Clojure, Prismatic Schema, Components, and ClassNotFoundException

I’ve been using clojure to some extent for about 3 years now. Up until very recently, I haven’t had one single reason to even look at the whole defrecord thing.

Recently, I came to the reluctant conclusion that sometimes things like type hints are good things. Usually when the functions that can benefit from them involve parameters that are just too complicated to work well with language standard type hints. So I finally broke down and learned me some Prismatic Schema.

Which, for the record, is wonderful stuff. Even though I haven’t used it for anything but code documentation . Yet.

One of my favorite parts is that it seems to work best with plain ol’ data structures like maps and vectors.

Somewhere in the same time frame, I started switching from Stuart Sierra’s Workflow Revisited pattern to his shinier Components library.

Which, for the record, is also great. It has drastically simplified a lot of my code. And made it glaringly obvious which parts of my start sequence are truly awful.

These two great tastes taste great together.

Until you need to refer to a record in a namespace that’s different from the one where it was defined. Such as specifying that your Session record in foo.session.core has a member that’s an instance of Database, defined in foo.persist.core.

That’s when the compiler starts throwing ClassNotFoundException. And, if you’re like me, life is miserable, because you just changed a whole slew of namespaces without carefully eval’ing each change because, hey, this is just grunt monkey coding. So you can’t even get to a REPL to try to figure out what’s going on.

This is a problem that’s plagued clojure since the day defrecord was introduced. The main result that google kept pointing me at is a stackoverflow question that boils down to “You have to import it.” Along with assurances that this had been in 1.4.

The import line was where the error was coming from.

The second major hint was to be sure to replace any – in the names with _, because this is java land. I was definitely doing that wrong (and this part is absolutely vital), but it didn’t help.

The magical equation (at least on 1.6.0) seems to be:

  1. Add a (:gen-class) to the ns declaration at the top of the namespace with the record that you want to use elsewhere
  2. In the ns directive for the namespace that’s going to actually use that record, add a reference to its namespace in the (:require) section
  3. Put the (:import) section below (:require)
  4. Double-check that you remembered to replace dashes with underscores.

There may be a slightly less magical incantation that I’m missing, but I haven’t been able to make it work any other way.

TODO: get on IRC and verify this.

xpra

I recently discovered this little gem. It took a little time and effort to figure out the basic setup, but it was totally worth it.

The website bills it as “screen for X”. Which doesn’t really mean anything if you aren’t familiar with screen. The short version is that you can connect to a remote machine, use windows that it creates on your desktop, and reconnect to that same session later from a different computer.

Which, by itself, is awesome.

For my purposes, though, it means that I can ssh into some VM running on the cloud and locally interact with windows that it creates.

Setting it up wasn’t quite as seamless as it seems like it should have been, but this *is* a fairly tricky problem.

In one configuration, I’m running ubuntu 14.04.1, which packages version 0.12.3. That seems to be the latest release version, so isn’t horrible. Most distros are much farther behind (big surprise, I know). The “stable” one that gentoo includes by default is 0.8.8.

Which (according to their website) is horribly flaky and incompatible and should never be used under any circumstances.

The web site is also a little vague on the “getting started” part. Install a relatively recent version on both client (your desktop, in this scenario) and the server (my VM).

On the client/local machine, run:

xpra start ssh:SERVERHOSTNAME:100 --start-child=xterm

Buried somewhere in the middle of the output (which probably includes a bunch of warnings about missing packages, though that isn’t what they look like) is a request for the password. Of the user from your local machine.

You really want to pip install numpy and PyOpenGL on the remote/server, if they aren’t already. Then the actual command on the client becomes:

xpra start ssh:remote_user@SERVERHOSTNAME:100 --start-child=xterm

Which fails after login with an error that

sh: 1: /home/remote_user/.xra/run-xpra: not found

This error should actually say something along the lines of “run xpra on the server”. So ssh into there and run

xpra start :100

The number is just an identifier for the screen. You can pretty much pick whichever suits your fancy, except 0. If you’re going to have conflicts, you probably know more about this than I do anyway.

That should report that it’s entering daemon mode and tell you where it stores the error log (in my case, that’s ~/.xpra/:100.log).

You can list your currently running sessions (with their associated ports) with

xpra list

From this point, my gentoo install seemed like it was working, though it wasn’t actually doing anything.

On my ubuntu client, I got:

AssertionError: proxy/shadow-start: expected 1 argument but got 2

which led (thanks google!) to a ridiculous diatribe about how the developers are all horrible people for making such awful software publicly available. It’s vaguely amusing, as an example of how to not get help (though they were much nicer about it than I would have been), but I’m not going to promote flames or trolls by linking to it.

The solution to that particular problem was to edit /etc/xpra/xpra.conf, find the line that said

start-child = /usr/X11/Xsession true

and wrap quotes around the command to convert it to

start-child = "/usr/X11/Xsession true"

But I still wasn’t getting any windows.

xpra info

is another useful command. It spits out a ton of configuration settings. At the bottom, it has an entry for the number of active windows available.

Mine shows 0, apparently because the –start-child command to xpra start doesn’t work the way I expected.

So, back in the terminal where I’m ssh’d into the ‘server’, run

DISPLAY=100 && tmux new -s PICK_A_NAME

and then launch interesting pieces (like, say, an IDE) from in there.

 

jzmq and Error 0x16

I’ve been dabbling with JNI for a while, trying to get Curve Encryption working inside the java binding for 0mq.

Everything looks like it’s working nicely, until I try to actually set the server’s private key (which, honestly, is step 2…but I’m pretty thrilled that I finally got the monster to build at all).

Setting that socket option fails: “Invalid argument(0x16)”. I spent some time digging through my code trying to figure out how and when I might be supplying that. Then I verified that the same call basically works fine with the python bindings (though I didn’t dig in to look at details). And *then* I resorted to google.

Where all the results were about Storm, and rolling back to some really ancient version they forked back before jzmq had released any code. The general consensus seems that the fix is to roll your 0mq version back to something really obsolete, like 2.1.7.

All this was barking up the wrong tree. 0x16 is the hex code for EINVALID. I should have paid more attention to the actual error message.

I haven’t tracked down the actual problem yet, but the point is that this is an *extremely* generic error message/code.

Lisp for Beginners

One of the few online forums I find valuable enough to make it a point to really follow these days is the clojure mailing list.

I recently realized that there’s one recurring theme that really irritates me: the “make it easier for beginners” thread.

This sort of thing always seems to come from someone who either a) is learning to program or b) is teaching a beginners’ programming class. You get a few belly aches from experienced programmers who are trying to make the conceptual leap into either lisp or java, but those are pretty rare.

I ran across one recently that was complaining about paredit mode. Which, admittedly, threw me for a loop when I ran across it the first time. I couldn’t understand how brilliant and wonderful it is until I’d written enough lisp to appreciate it.

Apparently, it’s the default mode in LightTable. Which, given its background, should be great as a beginner’s tool. The fact that it takes not-insignificant effort to switch to something that’s more immediately obvious to newbs (the “normal” thing where the closing parenthesis appears only and whenever you type it) doesn’t immediately and obviously counter the huge advantage you get by working with the actual tree structure of your code in a way that simply forbids creating unbalanced parentheses.

It’s something you need experience to appreciate.

I think that a lot of lisp is that way.

I won’t try to re-hash or top Paul Graham’s Blub Paradox. I’ll just state that I’ve seen it in action and fought many losing battles against it. Programmers generally don’t like getting pushed out of our comfort zones any more than anyone else. Many of us may intellectually accept that it’s a really good idea to learn a variety of languages for the sake of having a variety of tools in our box, but most of us never really have time to explore more than a sprinkling to any degree.

Even when we do, that sprinkling is likely to be C, C++, C#, Java, and javascript. Maybe python for the really adventurous. If you happened to take a beginner class centered around lisp, it was almost definitely scheme, and it probably left a foul taste in your mouth.

Arguably, scheme may be the perfect language for introducing beginners to the important concepts and moving straight into getting stuff done. I think python’s probably easier and better, because it has so many more batteries included, but that could very well just be my C++ bias. Either way, that foul taste is almost inevitable: if you try to graft useful appendages onto the flawless diamond that scheme is, you wind up with a ball of mud that resembles common lisp.

I’m starting to believe that lisp simply isn’t a good choice as a beginning programming language. You almost need to spend years coping with all the garbage that other languages dump on you before you can begin to see the benefits of a nice, clean language with a very regular syntax.

Even then, common lisp seems likely to be more attractive at first glance than something truly alien like clojure. The object orientation may be screwy, but it’s undeniably powerful. And any decent programmer worth his salt ought to be able to cope with mutating state.

Right?

I didn’t give clojure a fair shake until one road trip where I started designing my own language that I was planning to implement in common lisp. Three hours (and about 150 miles) later, I realized that clojure already had pretty much everything I wanted. Immutable state and ties to the JVM seemed like relatively small prices to pay for all those features I thought were really important to me.

I didn’t realize that those were two of its biggest strengths.

It’s tough enough to convince seasoned programmers that it’s worth exploring a language with screwy syntax and alien concepts. I can understand the temptation to inflict that on beginners, but it really seems to do more harm than good.

I know that I look back on my first language with contempt and disdain. (This is fair, since BASIC really isn’t good for much beyond teaching beginners).

Then again, there *are* the rare few who will recognize how much easier they had it back in the beginning and will be willing to try to push the rest of us to at least take a look at what we’re missing. But most of them will have forgotten the power they wielded as they grind down their spirits at the wheel of some enterprise language.

It’s probably kinder to just spare them that suffering and let them discover the worthwhile languages on their own. The ones who truly care about their craft will. The rest will keep going back to the enterprise languages no matter what.

Man Cave to Nursery

There’s really only one space available in the Nuthaus for Rooby to hang out. It’s the room that used to be my office.

So we’ve spent the past couple of weekends finding other places for my “absolutely vital” stuff. Most of which has been sitting around in boxes for the past few years anyway.

You might possibly be surprised (if you didn’t know me) to learn that at least half of it was garbage. Or, at least, prime candidates for recycling. Which still wound up in the land fill…Austin’s recycling schedule just does not work well with this sort of activity.

I found some interesting artifacts. It looks like Grandma filed all of the letters to her by family tree. And I seem to have inherited everything from Mom. Maybe it was supposed to just be the letters that were about me, but I’m willing to bet that no one bothered to trim them down by subject.

I was more than a little amazed at some of the stuff my mom had packed away for me. There were samples of our 4th grade newspaper. A box of 10-d nails. A Christmas tree ornament shaped like a train engine stamped “James”. A package of glue sticks. A knitted baby mitten.

This room feels empty to me now. I think it’s screaming for Rooby’s stuff. I wanted to start moving that in.

But Laura wants to wait until I really have it empty, and I can understand that. We’re planning to move the rest of my stuff out next weekend, but now I’m starting to feel the pressure too.

Then again…I tend to get really fidgety when my computer’s turned off. So maybe we’ll wait.