Recently it came to pass that my company needed to update its Platform Framework and Applications, which was based on Ruby 1.8.7 (Ruby Enterprise Edition). New features required new gems which required newer versions of existing gems which either broke existing conventions/APIs or were no longer supported on 1.8.7. Then we learned REE had reached End of Life. Nuts!
I’ll save a deeper analysis of the why for a separate post. The short of it is that we were already faced with a full regression test impact to get the changes we needed, so it made sense to invest a few more weeks and bite the whole bullet: hop onto the next Ubuntu LTS release in production, upgrade all of the (mature) gems, and upgrade the VM to the now-stable Ruby 1.9.3. This was a Big Deal.
How to go about this, then? Well, the ChangeLog could be enlightening, and of course there’s tons of blog posts about 1.9.3 itself. But when it came to actual conversions, the wisdom was surprisingly scarce. My sense is that the Ruby community is still relatively small overall, only a small subset of the community has ever (successfully) built stable, mature, long-lived platforms on the language, and maybe only a tiny fraction of them have ever had to work through the EOL of their particular VM and get on a new version.
So, here’s the skinny on converting a large, mature Ruby codebase from 1.8.7 (REE) to 1.9.3. While it’s not exhaustive, it will be representative of the sort of things to look out for. To be fair, a number of the problems encountered were due to “liberties” we had taken with the syntax – a core value of Ruby, and a big reason why so many love it. But that’s harsh criticism that misses the practical reality of a team of developers working hard and fast through the evolution of a large codebase over an extended period of time.
This post is by no means an indictment of 1.9.3. No, 1.9.3 is Better, and in many cases also Faster. But it ain’t perfect, and you’ll do better if you have some idea of what to expect.
Syntax
First up, syntax. While some of this was expected, some things were pretty subtle (e.g. array splat behavior).
rescue
gains higher precedence thannot
1 2 |
|
- parenthesis generally required where precedence gets confused
1 2 3 |
|
- can’t use multi-value assignment as conditional anymore
1
|
|
- can’t use instance variables as formal block params
1
|
|
- splat Array/List deref remains Array regardless of # elements
1 2 3 |
|
- splat behavior exception: when assigning to list of lvalues
1 2 3 |
|
Encoding
Anyone who has worked with non-ASCII character sets in 1.8.7 will know of Ruby’s
shortcomings when it comes to other encodings and codepages. 1.9 improves the
situation significantly, and actually deprecates the iconv
library.
- embedded UTF-8 chars/strings require explicit file-wide encoding
- insert
# encoding: UTF-8
at top of each affected file
- insert
- manual “guards” against invalid byte sequences require forced binary encoding
We had several cases using a Regex
to check a String
for invalid byte
sequences, as in user-provided inputs or validating email addresses. It
simplifies to this:
1 2 3 4 5 |
|
Note: You’ll get a SyntaxError
exception instead if you run this in irb
.
$KCODE
is gone;ruby -K
is not its equivalent -__ENCODING__
is__ENCODING__
only set by shell locale encoding (*nix) or codepage (Win32)
1 2 |
|
1 2 3 |
|
Beware: Win32 Ruby gets unstable quickly when default codepage is UTF-8 (aka crash)
- (Win32)
win32console
completely breaks$stdout
encoding- given the right encoding, UTF-8 chars will display to
$stdout
correctly - after loading
win32console
, UTF-8 is garbled again
- given the right encoding, UTF-8 chars will display to
1 2 3 4 |
|
Threading (Win32)
Thread
is no longer Green - it’s now a native thread
We wrap Microsoft Word with an asynchronous control channel using OLE and Eventmachine-driven AMQP, primarily for high fidelity split/combine & track-changes operations on Word documents.
Each AMQP message spawns off in a new Thread
. With green threads in 1.8.7, we
were effectively isolated from the vaguaries of various COM threading models
underlying OLE. Under 1.9.3 however, each Thread
is native and thus needs its
own OLE object handle, otherwise you get a WIN32OLERuntimeError
exception with
the following message:
1
|
|
We found a good discussion on the topic and a potential solution involving
calling CoInitialize
/CoUninitialize
per-Thread
, however it didn’t solve
our problem.
Instead, we simply memoized the OLE object handle in Thread
-local storage:
1 2 3 |
|
Kernel
Kernel
method enumerators produce anArray
ofSymbol
, notString
Module
Module.constants
produces anArray
ofSymbol
, notString
Enumerable
Enumerable.map
behavior changes- implications on a loop that expects the yielded value to work like a
String
(e.g.value#[]
)
- implications on a loop that expects the yielded value to work like a
1 2 |
|
Exception
Exception#message
is now immutable
Many of our Platform Framework libraries wrap various vendor-specific
Exception
classes with our own, to keep our library consumers within their own
domain and to guard against coupling consumer unit tests to indirect
vendor-specific behaviors. In some cases we add our own prefix/postfix to
existing #message
strings. Can’t do that in 1.9.3:
1
|
|
Exception#message
now forces#to_s
on value
We had some cases where we wrap vendor exceptions with our own, and tunneled the
original Exception
in #message
. That won’t work by default in 1.9.3, so we
added this:
1 2 3 4 5 6 7 |
|
Hash
Hash
now natively respects insert order- e.g. beware web tests that compare stringified JSON results
- inserting into
Hash
while enumerating it raises an exception
1 2 3 |
|
Hash#merge
doesn’t useHash#[]=
anymore- e.g. stringify’ing things upon assignment doesn’t work with
Hash#merge
- e.g. stringify’ing things upon assignment doesn’t work with
1 2 3 4 |
|
Hash
enumerable methods now returnEnumerable
1 2 3 4 |
|
Symbol
Symbol#match
now exists but doesn’t work likeString#match
1 2 |
|
String
String
no longer directly enumerable, so#map
,#grep
etc is gone- but you do get
String#each_{byte,char,codepoint,line}
- but you do get
1 2 |
|
String#[Symbol]
doesn’t (silently) work anymore
1 2 |
|
String#[Fixnum]
returns char-as-string, not ordinal
1 2 |
|
String#include?
no longer takes aFixnum
(as ordinal)
1 2 3 |
|
String#strip
doesn’t work on non-ASCII whitespace
1 2 |
|
But using the POSIX character class does:
1 2 |
|
Binding
- getting an object’s
binding
requires#instance_eval
instead of#send
A good example is the pattern of using an OpenStruct
initialized with a Hash
(which provides method accessors for keys) as a Context object for template
interpolation systems like Erubis
.
This used to work in 1.8.7:
1 2 3 |
|
This works in 1.9.3 and 1.8.7:
1 2 3 |
|
Fixnum
Fixnum#to_sym
no longer (silently) works
1 2 |
|
Float
- fraction-less
Float
becomes aFixnum
through#to_s
(precision lost)- e.g. cucumber test with a table containing a
Float
like10.0
- e.g. cucumber test with a table containing a
1 2 |
|
BigDecimal
BigDecimal
problems were among the most difficult in the conversion. There
are differences in behavior between VM versions and amongst BigDecimal
representations within the 1.9.3 VM itself.
- BD init by
String
skips precision altogether
1 2 |
|
- BD compared to
Float
makes a BD implicitly fromFloat
1 2 |
|
- BD +
Float
makes aFloat
1 2 |
|
Float#to_d
ignores class-level precision limit setting entirely
1 2 |
|
Note: Float#to_d
comes from the bigdecimal
library.
Float#to_d
uses a different “default precision” thanBigDecimal.new
1 2 3 4 |
|
- BD invocation differs depending on init param type
This makes writing generalized code for handling a BigDecimal
overly
complicated.
1 2 |
|
A Practical Look at the Problem
The Float
vs. BigDecimal
issue is about precision and “true representations”
of Rational numbers vs. representations that are incorrectly assumed to be
precise (like Float
). The story almost always ends the same way: mixing
differing-precision numbers in math calculations is just a Bad Idea. Mixing
BigDecimal
and Float
certainly qualifies as this.
The problem in 1.9.3 is that different invocations of BigDecimal
also
qualify as differing-precision numbers. A String
-initialized BigDecimal
will skip any precision. Convenience methods like #to_d
will produce a
BigDecimal
using a different “default precision” than the normal #new
.
#to_d
doesn’t even respect BigDecimal.limit
.
Sure there are ways to compare the precision of a BigDecimal
, and it is
possible to write extra code to ensure same-precision computation and
comparison. But on a practical level that’s terribly obnoxious and simply not
going to happen. Your average application developer just doesn’t care, and
shouldn’t. It’s hard enough to avoid mixing Float
and BigDecimal
, and
getting that right should be enough. Almost.
A Practical Solution
After adhering to the “don’t mix fractional types” rule, we were left with a
bunch of broken tests around an ORM property type designed for persisting a
BigDecimal
. In the end, those all resolved to this simple, new behavior in
1.9.3:
1
|
|
So a BigDecimal
is being compared against a Float
– big surprise that a
mixed precision comparison causes problems! But practically speaking, it’s
simply unrealistic for it to happen any other way: no one’s going to adjust
every line of their code and tests to detect+match precision, no one’s going to
update fixture loaders to know about and adjust to ORM-specific representations
of precision/scale-limited rational numbers, no one will investigate+adjust test
harnesses like Rspec and Cucumber/Gherkin to understand when a rational should
be a Float
vs. a BigDecimal
and at what precision. Realistically, fuck all
of that.
Instead, we just changed the implicit type conversion when comparing against a
Float
to cast to a String
first:
1 2 3 4 5 6 |
|
Then the test cases start working again, without having hacked any representation’s real scale/precision:
1
|
|
And just like that, the old behavior. Awesome.
FYI we used 80.784
because it was a (calculated) value in some tests that
triggered the BD-related problems. There’s nothing else special about the
value.
Closures
lambda
rules about arity have changed
1 2 |
|
This manifested as a result of an old rspec gem (1.2.9) that wasn’t getting
upgraded with the mix. Test it
declaratives without closures would raise a
special NotImplementedYetError
exception, automatically marking them as
pending. It did so using a lambda
, which it later called with an argument –
which doesn’t work anymore in 1.9.3. Swapping it to proc
with a monkey patch
solved that.
- initializing a
proc
with aproc
no longer catches return-from-inside-closure
This was a hack originally invented to wrap our ORM’s transactional layer with our own, in order to provide better commit/rollback semantics and deferred AMQP message delivery (outside the transaction window).
In general, we try to avoid obfuscating our code paths with too much nesting by
early-exiting from subroutines whenever possible, usually by calling return
.
And sometimes those subroutines end up being transaction closures.
From a pedantic POV that’s a terrible idea – closures aren’t methods, they have slightly different semantics, blah blah blah. Practically speaking, who cares. It should Just Work. So we made it work. And then it totally broke in 1.9.3.
It essentially resolved to this:
1 2 3 4 5 6 7 8 9 10 11 12 |
|
The idea of adhering to the Principal of Least Surprise around return
from
closures wasn’t unique to us; Sinatra had the same concept too, just using a
different mechanism: unbound methods.
So we use that now, and the method works like a charm on both 1.8.7 and 1.9.3:
1 2 3 4 5 6 7 8 9 10 11 12 |
|
Hooray!
Library / Gems
FasterCSV
renamed toCSV
and included in the core Ruby distribution
json
(v1.5.4) included in the core Ruby distribution- beware non-rubygems
$LOAD_PATH
manipulations – Ruby’s libs always come first
- beware non-rubygems
soap4r
no longer bundled in the core Ruby distribution- old
soap4r
gem is incompatible with 1.9 –use soap4r-ruby1.9
- old
sha1
library is gone- use
digest/sha1
instead
- use
ruby-debug
gem et al. renamed todebugger
gem et al.
$ENV['CWD']
not in$LOAD_PATH
by default anymore
- XML parsing is more strict
- can’t embed dash-dash (“–”) inside XML comments
- YAML parsing is more strict
- bare regexes need to be quoted
- bare regexes containing “escaped”-style character classes require single-quoting
- YAML parsing also got a little more convenient
Time
/DateTime
/Date
values serialized asString
are auto-converted to native type
Net::HTTP::Post.set_form_data
now usesURI.encode_www_form
This subtly broke signature calculation on our Oauth consumers vs. our provider.
Oauth consumers (oauth
gem) use Net::HTTP::Post
while Oauth providers
(oauth-provider
gem) use Rack::Request
, but param collection and
serialization for signature calculation was effectively equivalent in 1.8.7.
Now Net::HTTP::Post
uses URI.encode_www_form
, and that produces a
different result when given a param key with a nil value. Thus signature
calculation on the consumer would differ from what the provider arrives at, and
request authentication would fail.
Our “fix” was to just guard against nil parameters before making requests.
1 2 3 4 5 |
|
Evil!
open-uri
is a rainbow-hating, puppy-murdering Nazi who wants to make your life harder
If the userinfo
portion of a URI
is set, it raises an exception:
1 2 3 4 |
|
This turned out to be one of those tired, irritating pedantic engineer vs. practical utility arguments where the pedant won. Hooray for pedantic idiots!
Conclusion
Well, that’s all I got for now! Check out Harvest’s Ruby 1.9.3 upgrade blog post for some additional pitfalls to watch out for.