notkeepingitreal.com

software, business, and other business

Clojure Conj 2013

These are some notes from Clojure Conj 2013, where 370 actual or aspiring clojure developers hung out for three days.

data.fressian

The data.fressian serialization/encoding library (to be released today) is pretty much edn (same goals, including cross-language), but designed for performance (it’s binary).

core.matrix

core.matrix is an array programming as a language extension for Clojure, with a numerical computing focus. It’s in the same family as R.

Schema

“Some of what makes Clojure great to write can make it difficult to read.” Doc-strings can help a bit, but most of the downsides of comments (principally, they get out of date) also apply. Schema is a library developed at Prismatic out of a desire to make clojure code more maintainable for new and experienced clojure developers alike. It allows the author of a function to specify the inputs and outputs in shareable, all-data, schemas. These schemas are great for making code more readable, but they can also be used to do runtime checking to ensure that your code is doing what you intended (you can turn this on by function, by namespace, or globally, and it is great when paired with automated tests). At Prismatic, the rule of thumb is that all functions meant to be used by other namespaces should have schemas. Schemas are not quite types (in fact, they are more flexible, since you can define your own structures and predicates) and not quite contracts. This is one of the things that is most interesting to me in terms of something to take back and directly apply to our production codebase.

Illuminated macros

These guys have been writing some code to help make macros less difficult to write and use. This is something that looks like it might get back into clojure core.

Harmonikit

Rich Hickey wrote a harmonic synthesizer and an iPad UI, since “you need emacs for making music like you need it for having sex. And, for the ubergeeks in the audience: that is NOT AT ALL.”

Cascalog

Cascalog is a datalog DSL written in clojure that helps you write map reduce jobs. It sits on top of cascading, which in turn sits on top of hadoop. Recommended book: The Design of Design (Fred Brooks). Cascalog is another nod to logic programming.

Hand tools

Tim Ewald gave a very fun talk on why he uses hand tools:

  • Simple to the point of primitive
  • Great precision and control
  • Flexibility to make anything
  • Connects one to the work, and the effort to the results

He noted how the rise of automation in woodworking has required trade-offs, including that design and proportionality have very often been trumped by cost concerns. In software, too, where we love automation, we would do well to be aware of potential downsides:

  • Don’t spend hours trying to make something fit into your favorite super-high-level DSL, when a perfectly fine alternative exists at a lower level
  • Just because your development environment isn’t aware of something (like a message bus) doesn’t mean that that thing is not valuable
  • Our build tools (maven, leiningen) do lots of things for us; do we have enough knowledge to do what they do when their high-level abstractions break down?
  • If you want to build something unique, use simple tools

“The more you automate, the less deeply you see. Be mindful when you automate; know your tools.”

propaganda

Propaganda might be a good choice for you if you want to model a problem declaratively (like core.logic), and it particularly excels in arithmetic applications, and in its ability to show you the path the program is following to find the solution.

vert.x

vert.x is a web framework built on netty that offers an async, reactor-based run loop (like node and goliath) for efficiency, though it runs one reactor per core, not per process. It offers first-class support for a variety of programming languages. One of the main features of its architecture is an event bus that sits in the middle of the other components, keeping the subsystems very decoupled. The browser can also participate with the event bus, getting streamed events via SockJS, which gives you bi-directional communication via websockets, or various fallbacks.

liberator

liberator is a clojure library that can be used to implement rest web services (mainly has ring in mind) on the server side. If you have lots of if/else’s in your resource to handle edge cases and return various http codes, you may benefit from using liberator’s DSL

zeder

Zeder is a production rules system that borrows some of the good datamic conventions and has some overlap of utility with core.logic. Fogus recommended Out of the Tarpit, calling it the “decomplecter’s manifesto.”

To the Moon

Apollo, the mission to send people to the moon, had an arbitrary goal and deadline (just like many software projects). “We choose to go to the Moon in this decade and do the other things, not because they are easy, but because they are hard.” -JFK. Takeaways:

  • Little things can kill you
  • Followship: the people in the mission control center didn’t yell at Neil Armstrong and tell him to stop when he choose to take the Eagle ship off autopilot and land it himself (he had observed that it was on path to land in a crater full of car-sized boulders).
  • The moon landing is the cultural heritage of technologists like ourselves. Nothing is impossible, and we have to power to make it happen.
  • Russ Olsen is quite an amazing speaker.

Scheme and functional programming workshop 2013

Scheme has been around for 38 years. The scheme community is interested in cross-polinating with the clojure community, so they gave away free passes to the 2013 scheme and functional programming workshop to those folks attending Clojure Conj, including me.

I wanted to set up scheme on my box to try to fit in. Thomas Gilray, one of the presenters, who was sitting near me, didn’t immediately say “yes yes” when I told him I was about to install mit-scheme, but indicated that he uses racket, which is apparently almost a superset of mit-scheme. I installed the dmg, since racket wasn’t available through homebrew, started up the DrRacket application, and had a functioning repl:

1
2
> "hello world"
"hello world"

Alexey Radul, Ph.D., MIT, and the keynote speaker, talked about propagators, which he had described in the art of the propagator, one of his major academic projects. He agreed that his work is similar to functional reactive programming (FRP). The rest of the day was broken up into three sessions.

Session 1 (Static Analysis)

Entangled Abstract Domains for Higher-order Programs

Shuying Liang talked about using an entangled abstract domain to prove when a list is bound.

Multi-core Parallelization of Abstract Abstract Machines

Leif Anderson researched whether static analysis processing could be sped up by using multiple cores. He concluded that, in the worst case, omega (infinite loop), multiple cores doesn’t speed up analysis at all, and in the best cases, with functions like factorial, the speed up can almost keep up with your number of cores (he got 7.x times faster, and only had 8 logical cores). The source is on github.

A Unified Approach to Polyvariance in Abstract Interpretations

Thomas Gilray explained that Abstract Interpretation is a general framework for Static Analysis that makes it easier to model computation, trading off precision (your abstraction isn’t exactly the same as the program being analyzed). Here’s a paper on a similar topic to that presented.

Session 2 (miniKanren)

miniKanren is a relational, declarative programming language, and it inspired clojure’s core.logic library.

mu-Kanren: More with Less

Lispers love infinite lists. Jason Hemann pointed out, quoting someone, that “Infinity is only a problem if you try to print it.” His project, micro kanren is 39 actual lines of scheme code, and needs a clojure port.

rKanren: Guided Search in miniKanren

Cameron Swords demonstrated a miniKanren implementation that allows us to get back the answers we want first, by ranking clauses.

Session 3 (General Scheming)

lambda*: Beyond Currying

Jason Hemann argued that currying is pragmatic, and elaborated on a way to implement it in scheme with a library he wrote, lambda*.

Automatic Cross-Library Optimization

The concept of inlining procedures (inline expansion) came up here, and it had come up earlier today as well. This is a common compiler optimization where the code in procedures is shoved into the calling procedure to speed up runtime performance. Some procedures in scheme can be inlined, and some not. Andrew Keep concluded that using library groups offers the best speedups, but when you can’t write your scheme program with library groups in mind, you will often still get some speedups from automatic cross-library optimization.

R7RS Scheme Standardization Update

Will Clinger reported that the current direction is to split the scheme language into two languages: 1) a small language suitable for education and 2) a larger language (more features) suitable for industry. Will said that the small language spec has been approved, so the first working group is finished with its task. Working group 2 is still on the task. There’s more info at scheme-reports.

All in all, I was glad I attended. I realized that I should read The Reasoned Schemer. The workshop was extremely academically-focused. I was able to get a fix of pragmatism by chatting with Gary and Paul (revelytix) over a break about leiningen plugins and polyglot clojure/java applications. We also wondered why 1) ruby has three types of functions and 2) if the only reason you have to .call ruby Procs is because of ruby’s optional parentheses.

Clojure with Vim

These notes are intended to be used alongside this screencast, and describe some aspects of setting up and using a clojure editing environment in vim, if for some reason you don’t want to use emacs. It is very unlikely that I would ever switch to emacs, but if there is a killer feature of emacs/clojure not found here that you cannot live without, I’d be interested in knowing about it, so that I can cry myself to sleep/try to figure out if anyone in the clojure/vim community (not just for the VimClojure plugin any more) has replicated the functionality.

Assuming you don’t want to clone my dotfiles repo, which are pretty much skwp’s dotfiles but with some extra clojure stuff, here’s the important parts for clojure:

Install

1
2
let g:rainbow_active = 1
let g:rainbow_operators = 1
  • Make sure your Clojure project follows leiningen conventions, possibly by generating your project with leiningen:
1
lein new todos

Using this setup

In a terminal window, start up a lein repl:

1
2
cd todos
lein repl

In a second terminal window, open up your project in vim.

There’s lots of ways to open the file you want to edit in vim. I like using ctrlp, which has nothing to do with clojure.

My clojure source files have sweet syntax highlighting and indentation from vim-clojure-static, but pretty much all of the functionality comes from vim-fireplace.

These are the features from fireplace I use a lot (borrowing heavily from tpope’s docs here):

  • cpp to evaluate the expression under the cursor

  • K to look up the docs for the symbol under the cursor

  • [d to look up the source code for the symbol under the cursor

  • gf to “go to the file” corresponding to the namespace under the cursor

  • Require! to require the current namespace with :reload-all, which also reloads dependencies

  • :help fireplace for the rest

There’s lots of other info out there regarding using vim to write clojure, including vim docs on clojure’s site itself.

On new (to me) blogging platform: Octopress

There was a serious security hole affecting ruby on rails announced the the other day, and this blog was affected. It was also affected by hundreds of other security holes, since I hadn’t upgraded my blog engine, typo, since 2006. :)

Since attackers could have executed arbitrary ruby code on my shared hosting instance, I decided it was time do something, so I checked out a comparison site, discovered that octopress is the most popular ruby blogging system these days, and decided to switch. So far so good; thanks for the migration tools, octopress!

RubyConf talk: William Bereza: Enhancing Embedded Development with Ruby

This guy was creating an embedded system, and basically just wanted to test out ruby. Hard to blame a guy for that.

Back in the day, we had low-level languages, slow computers, and less tools. These days we have high-level languages, fast computers, and mad tools. So programming is easier except on embedded devices, which remain in a scenario similar to that of the aforementioned “back in the day”.

If only there were a way to bring testing, continuous integration, and other best practices to the embedded world..

Maybe ruby could help.

They had a couple rake targets: rake test:system and rake test:units. There was black box testing of the hardware, and mocking of various bits of the hardware.

They were able to link in with the continuous integration system used by the rest of the company (for rails projects, etc), since the embedded project build task was written in rake.

Results of using this rubyized system:

  • DRY C code
  • readable system code
  • readable system tests
  • tests!
  • customers were happy
  • developers were happy
  • lovefest and a half

RubyConf talk: William Bereza: Enhancing Embedded Development with Ruby

This guy was creating an embedded system, and basically just wanted to test out ruby. Hard to blame a guy for that.

Back in the day, we had low-level languages, slow computers, and less tools. These days we have high-level languages, fast computers, and mad tools. So programming is easier except on embedded devices, which remain in a scenario similar to that of the aforementioned “back in the day”.

If only there were a way to bring testing, continuous integration, and other best practices to the embedded world..

Maybe ruby could help.

They had a couple rake targets: rake test:system and rake test:units. There was black box testing of the hardware, and mocking of various bits of the hardware.

They were able to link in with the continuous integration system used by the rest of the company (for rails projects, etc), since the embedded project build task was written in rake.

Results of using this rubyized system:

  • DRY C code
  • readable system code
  • readable system tests
  • tests!
  • customers were happy
  • developers were happy
  • lovefest and a half

RubyConf talk: Matz: Saturday Keynote

Language matters or not?

Twitter made a 1000% performance gain with mysql, solaris, and dtrace. No language change. So if many languages provide similar feature-sets, what is the difference?

Attitude

Python believes in a “one true way,” while ruby believes in diversity.

Ruby is doing well – Martin Fowler of thoughtworks was telling me that ruby is making his life difficult, since if thoughtworks employees can’t use it, they quit. :) And that seems great, since there is business value in fun; after all, there is lots of gain in programmer productivity based on enthusiasm.

Ruby 1.9:

  • Major incompatibilities include encoding jank and block argument restrictions
  • Minor incompatibilities abound, including File.exists? [which some guy had asked about the previous night]
  • Lots of new features – he showed off external iterators

Languages (and empires) go through cycles:

  • Humble beginning
  • Success
  • Pride
  • Conflict/War
  • Drop to nothing

Ruby is somewhere between success and pride, so at least we know where we’re headed. :) We will eventually go enterprisey to please the “suit people” – yeah, I know, I don’t want to either. :)

We’ve got such an awesome, nice, smart community. Let’s change the world!

RubyConf talk: Kyle Maxwell: JRuby in the Wild

Jruby is already kind of an easy sell in big companies. It’s the quickness of rails with the familiarity of java.

But

I’m not trying to to sell JRuby to enterprises [well clearly, or else he wouldn’t be giving this talk at rubyconf].

JRuby can work well, even for startups – it’s just about choosing the best tool for the job.

Why might one choose JRuby?

  • performance
  • library support (in fact, we wanted Lucene for search!)
  • extending the language without using C

Getting started:

  • easy install
  • don’t use textmate to code java
  • casting can be tricky (java needs arrays to have only one type of object in them, ruby not so much)

RubyConf talk: Jay Phillips: Next-Gen VoIP Development with Ruby and Adhearsion

Adhearsion is not a rails plugin. Rails is certainly hot though. After all, the web is hot. Telecom, at least historically, is not at all hot.

Enter VoIP.

Asterisk, the defacto standard, has a fairly tough-to-get-into codebase. Maybe adhearsion can do something.

Your phone is actually a universal remote. There are mad possibilities for controlling doors, video games, and all manner of electronics.

There are lots of fun hijinx you can get into: spoofing caller id, locking out automated telemarketers, integrating with instant messaging, ETC.

Adhearsion is open source software that abstracts away this technology with ruby in an extensible, easy to understand way. “Weasels have destroyed the phone system.” Word!