lysergicjava

oh, technology

Menu Close

Month: August 2008

Closing the book on closures

I keep getting questions about closures, mostly from people who know me and my YAGNI attitude.  “I don’t get what’s so great about them.  What can you do with closures that you couldn’t do otherwise?”

Answer: Nothing.

Well, that’s only partly an answer.  It’s true, closures aren’t magic.  There’s nothing you can do with them that you couldn’t do otherwise, but there is something great about them.  For example, in Groovy, you could do something like this:

def sql = Sql.newInstance("jdbc:mysql://server/database", "sa", "pword", "Driver")
sql.eachRow("select foo, bar from foobars") { row ->
  def foo = row["foo"]
  def bar = row["bar"]
  print "Foo = $foo and bar = $bar"
}

Here, Groovy’s Sql class has a method called eachRow that accepts two parameters: a query, and a closure to execute for each row in the result.  Functionally, this is the same as:

// assume you've got your db connection setup
ResultSet rs = conn.execute("select foo, bar from foobars")
while rs.hasNext()
{
  def foo = rs.getString("foo")
  def bar = rs.getString("bar")
  print "Foo = $foo and bar = $bar"
}
rs.close()
conn.close()

…and so you may be forgiven for thinking closures are much ado about nothing.  After all, in both cases, all you’ve done is open a connection, execute a query, iterate over the results, and do something with each row.  Not a big deal, right?

Well, while functionally the two code samples are equivalent, the closure example reveals something special.  See the last two lines of the non-closure example, the calls to rs.close() and conn.close()?  You don’t need those when calling Sql.eachRow because it does it for you.  The designers of Groovy, in essence, recognized that when you’re dealing with databases it’s real common to open a connection, execute a query, iterate over the results, and do something with each row.  Oh yeah, and clean up the resources afterward.  Now, with repetitive things it’s a good idea to encapsulate the invariants (opening, iterating, and closing) and abstract what varies (the queries and what you do with the results).  The query’s already abstracted as a SQL string, so let’s abstract the code that processes the results and we’re good to go.

There’s something else.  What we’re doing when we pass a closure to the eachRow() method is metaprogramming — partially specifying, at runtime, the behavior of another object.  But we’re doing it responsibly; we’re not just clobbering or overriding some behavior, we’re fulfilling a contract.  Of course, without closures, you could accomplish the same thing by using, say, the command pattern.  (More or less this is how the Groovy runtime does it behind the scenes, anyway).  But the closure paradigm is cleaner and more convenient.

Which gets to the heart of it: using closures properly results in cleaner, more elegant code, and that’s all.  But isn’t that what you want, anyway?

Hacking Eclipse plugin configurations

Reading Neal Ford’s generally excellent book The Productive Programmer, I experimented with his multiple plugin configuration hack. In short, creating multiple plugin configurations in a single eclipse install allows for a team to keep their plugin configurations in version control, so everyone has exactly the same configurations as anyone else on the team. No more “works on my machine but not on yours” weirdness. You can even manage them on a project-by-project basis, which is good.

IT Stamp of Approval

How many times have you heard this?

But there are two aspects of working with multiple plugin configurations that are strange: specifically, creating and deleting them. In order to create an additional plugin configuration, you have to:

  1. Create a folder to hold the configuration. It must be named “eclipse” and it must not be in Eclipse’s directory structure.
  2. Within your “eclipse” folder, you have to make an empty file called .eclipseextension, and two empty folders, features and plugins.

As Ford points out, Eclipse (inexplicably) won’t do this for you. You have to do this by hand. It’s not hard, just strange. At least this way you have more control over where the folders and files are located. (As I have multiple Eclipse installs, I used nesting to keep track of everything. I created a top-level folder called “eclipse-configurations”… under that, I made another folder for each of my named installs, and under each of those, I placed the “eclipse” folder as mentioned above. So, the versioned configuration for my “xquery” install of Eclipse is at /Users/haren/eclipse-configurations/xquery/eclipse.)

From there, you can go to Help -> Software Updates -> Manage Configuration and add your configuration location(s). Then it’s a simple matter of installing your plugins to the config locations desired. You can then enable and disable multiple plugins as a group, switch between versions, etc. It’s very handy.

But I’d mentioned that there were two strange things about the process. Creating additional configuration locations was one, deleting them was the second. Just as Eclipse gives you no love in creating them, it makes it even harder to get rid of them.

Let’s say, for example, that you’ve added your new location as an extension not to the top-level list, but as an extension to an extension. (Yes, you can do this.) But let’s also say that’s not what you wanted. Well, you can disable your extension-within-an-extension, but you can’t get Eclipse to ignore its existence entirely. If you then try to add it to the top level, Eclipse won’t let you, complaining that you’ve already added it elsewhere. Arg.

Well, there’s a way around that, too (but Ford doesn’t mention it). Under ${ECLIPSE_HOME}/configuration/org.eclipse.update there’s a file called platform.xml. Up at the top there are “site” nodes, and one of those will be your offender. Delete the bad guy and restart Eclipse. Now you can place your configuration elsewhere. (Or, you can just change the path in the node).

Anyway, as noted, there’s a lot to gain by using multiple plugin configuarations, once you get around Eclipse’s strange reluctance to make it intuitive. Happy hacking!

Now we’re just getting silly

There’s an old beef in this industry regarding hiring practices — specifically, unreasonable requirements in job adverts.  Back when I was fresh out of college I learned to just move along if the job ad contained a screwy writeup, an unclear picture of the job responsibilities, or (especially) a requirements section that made it sound as if they’d settle for no less than Linus Torvalds himself.

Not having been in the job hunt myself lo these past few years, I’d more-or-less forgotten about the chicanery prevalent in tech-industry hiring practices.  But as two colleagues of mine have recently gone through a job change, I’ve been getting a glimpse of current job ads, and if anything, it’s gotten worse.  Granted, not in absolute terms.  I remember job hunting back in the immediate aftermath of the dot-com bust.  Employers were asking for the moon back then.

But today, while the ads aren’t quite as nutty as they were say, circa 2003, there’s far less excuse. Conventional wisdom holds that (a) a whole lotta incompetent chowderheads got into this industry during the boom and (b) the bust, if nothing else, weeded them out.  Eventually.  We’ve had several years of sink or swim, dog-eat-dog competition for jobs, and the devil has taken the hindmost.  Right?  And that should go for HR people as well.  I mean, engineers have this bias against HR people in general, but I have to say that most of the HR people I’ve worked with pretty much had their act together.  (There was this one really really stinker-oo exception to this, but I’ll not name names.)  Maybe the HR biz hasn’t shaken out all the deadwood yet, or maybe it’s upper management making goofy decisions, or maybe we, collectively, as an industry, still have our heads in the clouds.  All I know for certain is that requirements lists like the following still exist:

Minimum Qualifications

* A minimum 10 years of software development experience, at least half of which has involved direct Java/J2EE web or internet projects.
* Strong knowledge of development methodologies and how they can be applied to the organization.
* Firm understanding of project plans, estimates, milestones, and the overall software development lifecycle.
* Firm understanding of security issues as it applies to web applications.
* Strong experience designing database schemas for applications, and analyzing database specific issues. PL/SQL programming a plus.
* Very clear verbal ability in describing technical issues in both non-technical and technical language, depending on the audience at hand.
* Strong trouble-shooting ability in resolving tough technical problems.
* Intimate knowledge in how web servers, application servers, and database servers work together.
* Strong experience in Java, C/C++ and Perl development.
* Firm understanding and hands on experience with diverse technologies such as XML, SOAP, HTTP, JDBC, JNDI, JMS, JMX and JTA.
* Familiarity with Struts, JSF, Tapestry, Spring, Hibernate, Axis, JDO, and WebWork.
* Experience installing, configuring and deploying applications in WebLogic, WebSphere, and/or JBoss.
* Experience installing and configuring Apache Web Servers.
* Experience with unit testing.
* Comprehensive knowledge of Quality Assurance and how it fits into the development process.
* Proficiency in drawing system diagrams of large, multi-tier web applications from front to back, including networked and clustered components.
* Expertise in UML and recognizing which elements are applicable for a given project.

Sorry, guys, but your candidate doesn’t exist.  Just some of the middle-of-the-list throwaways like “Strong experience in Java, C/C++ and Perl development” alone are absurd to ask for, leastways for any reasonable definition of the phrase “strong experience”.  I’m quite experienced with Java, and I’ll go out on a limb here and say that I cut quite the dashing figure with C++ five years ago; I’ve not used it since.  Does this make me “strong” in C++ nowadays? Hell no.  Same applies for you, too.  I don’t care who you are, if you’ve not used C++ in five years you’re no longer an expert.  I do not know, nor have I ever met, anyone, anywhere at any time, who has even given me the impression that they are concurrently “strong” in Java, C++, and Perl.  As a friend of mine put it, “Being ‘strong’ in a tech skill is akin to being ‘strong’ in an athletic pursuit.  You cannot be an olympic level swimmer, boxer, and weightlifter all at the same time.”

Add to that other demands like “Firm understanding and hands on experience with diverse technologies such as XML, SOAP, HTTP, JDBC, JNDI, JMS, JMX and JTA” and “Familiarity with Struts, JSF, Tapestry, Spring, Hibernate, Axis, JDO, and WebWork” and you’re in cloud cuckoo land, if “firm understanding” and “familiarity” have any meaning at all.  Top it off with the rest of the above list and you’ve weeded out… yep, 164% of the population.  I don’t know how, exactly, but the numbers don’t lie.  And keep in mind, as the posting makes clear: these are minimum qualifications.  To me, “minimum qualifications” describes the sorts of things you should know, in order to prevent the interviewer from suing you for wasting his time, if you don’t know them.  That covers much less ground than what they’ve listed.

Look, I get it, guys.  You want top-notch personnel.  You want people who will be able to deal with the alphabet-soup world of modern development.  You want people who can adapt and learn.  Fine.  Just do us a favor and ask for that, in reasonable terms:

Minimum Qualifications

* Smart, and gets things done.  (Thanks, Joel!)
* Proficient in one of the following: Java, C++, or Perl.

There.  Was that so hard?

The fun isn’t limited to software engineers, though.  There’s this gem, courtesy of our friends at Sony.  The position: “Sr. Web Site Technologist”.  The job tasks:

1. Create and maintain content on internal company web (“intranet”) servers for the IT department and selected other customers. This will involve everything from designing “look and feel” (styles), to creating new content (technical writing and editing) and reformatting of existing content. This also includes selecting, installing and customizing scripts, content management systems, and other server-side software to enhance functionality of the sites and verify web site integrity and eliminate broken links and web site content errors. Work with other departments to establish web site usage procedures and policies.
2. Create web-enabled applications by assisting product development staff in creating interfaces to web services for existing and future internal applications, departmental “intranet” services and other web/application interfaces. Provide technical support as needed to the SCEA online game community on a wide range of issues concerning web content management, web-based collaboration systems and related services.
3. Create, maintain and optimize web servers by designing, installing and configuring highly reliable 24×7 production web servers for internal and Internet services. Optimize performance of servers by analyzing web site traffic, and present regular reports detailing analysis of internal usage of the web sites and identifying needed changes. Automate the monitoring of web server health, performance, and security. Research new web technologies, and, as necessary, implement those new technologies into existing web servers. Recommend hardware and software purchases as necessary.
4. Configure and maintain related services such as DNS, email, FTP, etc. as needed to support web site development and deployment.
5. Travel occasionally as needed to support remote offices and server collocation facilities.

Whew, that’s gonna be one busy employee, but I suppose sysadmin/content producer/writer/designer/tech support gurus are a dime a dozen.  Should be no problem, guys!  Oh, but wait… the dreaded…

REQUIRED SKILLS.
PEOPLE: Strong experience with teamwork in Internet-related server operation groups, very good technical-level verbal and written communication skills in English.
PROCESS: Ability to participate in process definition for change management and problem resolution to support large-scale 7×24 systems operations.
TECHNICAL: Must have computer science knowledge adequate to understand and communicate complex system concepts to technical teams; beginning to intermediate Unix/Linux systems administration and scripting knowledge; previous experience as a UNIX/Linux and/or web site administrator; intermediate MS Office skills; previous experience designing web sites including page layouts and use of content management systems; experience hosting web content on Unix/Linux systems; intermediate level Internet and related networking expertise; and basic understanding of PlayStation games.
PLANNING/IMPLEMENTATION: Must be able to create and maintain project plans in collaboration with technical teams and management.
INDUSTRY: Should have Internet background sufficient to understand operations, security, and network planning.

Well, okay.  Apart from the “7×24 systems operations”, that’s not too bad… I suppose a lot of people could cut the mustard in such a ca… wait, what’s this?

Responsibilities:
Administer and support existing intranet sites, applications and servers in a Microsoft environment, including IIS and Apache
Knowledge and support of Stellent content management software
Knowledge of Digital Rights Management (DRM) or Digital Assets Management concepts
Administer and support SCEA’s Content Management System
Create and edit web pages and manage Company /Consumer Services / departmental intranet sites
Perform varied maintenance duties within the intranet environment
Assist with user education and training on CMS functionality, and on the use of our web page templates.

“Responsibilities” on top of Job Tasks.  Umm, ok.  They’ve basically recapped the more verbose version above, replacing the Linux/Unix world with the Microsoft world, but now they’ve added “user education and training” and “varied maintenance duties” to the mix.  You want someone with carpentry and massage therapy skills while you’re at it?  OH DEAR LORD there’s more:

Qualifications:
Expert knowledge of HTML, CSS, ASP, SQL, VBScript, Java, XML, JavaScript Perl, XML, XHTML, , Web Services, Windows Service. Understanding of PHP
Familiar with Photoshop and web graphic formats.
Experience with Windows 2000, 2003, Internet Information Server 5 & 6, Apache and SQL Server 2000.
Knowledge of Stellent Content Management and Digital Asset Management Systems.
Database: MSSQL, SQL Server 2000
Languages: C#, Java (Applet, ODBC)
Basic graphics / UI / usability design skills are desirable.
Strong web application development and design experience.
Strong foundation in object-oriented programming.
Strong debugging skills in an n-tier application.
Must be service oriented with excellent communication/people skills.

And now, my friends, we’ve sunk into Monty Python-esque silliness.  Aside from nonsense like “Database: MSSQL, SQL Server 2000” and “Languages: C#, Java (Applet, ODBC)” they want a sysadmin/content producer/writer/designer/tech support guru/training session facilitator/maintenence man with a strong foundation in object-oriented programming.  Oh, and you’d better be a good debugger, too.  And yes, that was all from one job posting.

Sony: you’re nuts.  Granted, I’m not serious in thinking that anyone at Sony really expects a candidate to know and do all that.  And, one could argue, ads of that sort are just the product of some know-nothing hiring committee cutting and pasting stuff they don’t understand into documents they don’t read.  Let’s assume that’s true.  It’s not a good sign, for—and this is important—it’s clear that the organization that produced the above posting doesn’t know anything meaningful about the job, in particular:

  1. What the job responsibilities translate to in rubber-meets-the-road, dollars-and-cents terms for the company.
  2. The difference between a job well done, a job passably done, and incompetence.
  3. The number of people needed to do the job right.
  4. The caliber of people needed to to the job right.

They’re just like the guy who jumped on his horse and rode off in all directions.  If the organization either can’t or won’t (no real difference) understand the basic facts of the job, you do not want it, because they will treat you with the same arbitrary ambivalence as an employee.  Forewarned is forearmed and all that.

Oh, you crazy semantic web (YAGNI part 2)

Perhaps you’ve heard of the “semantic web“, aka Web 3.0, aka the Amazing Everything Machine.  Perhaps you haven’t.  Either way, it doesn’t make much difference — I’m going to boldly predict that it won’t happen.  (Now, I’m well aware that making bold predictions may make me look like a total fool later on.  I’ll take the risk.  In any event, I’m predicting it won’t happen on anything approaching the scale that its boosters envision.)  Briefly, if you’re new to this, the semantic web is a blanket term for a variety of technologies that will define, in machine-readable ways, the semantics (or meaning) of information on the web.  So for example, rather than just having some arbitrary string of letters saying “Posted by John Haren” (which any literate human can recognize as a byline) you’d have some non-arbitrary, agreed-upon standard string of letters saying “the author of this trash is none other than John Haren”. Well, why bother with that?  It depends upon who you ask.

To the average, sane person, it sounds kinda cool at first.  Having all the web’s semantics defined in a uniform, machine-readable way would, in principle, enable for much more intelligent usage of the web’s teeming content.  Just imagine!  Imagine if your computer understood what you meant when you asked it to search for people in your area with tickets to the opera/art gala/footie match and yet no ride to same.  Why, you could hook up in a twinkling!  Or, maybe you’d like to see what correlation, if any, exists between citing Yukio Mishima and Sylvia Plath as major artistic influences while in college and heavy smoking.  The semantic web is at your service, sir!  Oh, and the answer is yes, there is a correlation, and it is 1.0.  Less sarcastically, the semantic web does hold promise as serving as a vast inference engine.  If realized, it could revolutionize ways that information is organized, and the way that intelligence is gained from information.  What’s not to like about that?

Glad you asked. In a word: work.  As in, it’s too much work.  Here, all along, you thought the web was a huge, fathomless pile.  Well, that’s nothing compared to the pile of work necessary to implement the grand vision of the semantic web.

Grand, or merely grandiose?  The original vision of the semantic web is often (who am I kidding, often?  Always) attributed to Sir Tim Berners-Lee:

“I have a dream for the Web [in which computers] become capable of analyzing all the data on the Web – the content, links, and transactions between people and computers. A ‘Semantic Web’, which should make this possible, has yet to emerge, but when it does, the day-to-day mechanisms of trade, bureaucracy and our daily lives will be handled by machines talking to machines. The ‘intelligent agents’ people have touted for ages will finally materialize.”

Visionary!  [clap clap clap clap] How does he do it? [clap clap clap clap]

I guess no one’s stopped to ask Mr. Sir Tim what, exactly, he means by “the day-to-day mechanisms of trade, bureaucracy and our daily lives will be handled by machines talking to machines” but given that sentiment alone, one could make a more than reasonable case that reality has exceeded his vision already.  Nor is it clear what he means by [computers] “capable of analyzing all the data on the Web” — specifically, the analyzing part.  Doesn’t Google Zeitgeist do that?  Depends on what you mean by analyze, I suppose.  What Berners-Lee (or anyone) specifically sees happening has… well, not been made specific.  But it’s hard to get more hand-wavy than Berners-Lee when he said this:

“People keep asking what Web 3.0 is. I think maybe when you’ve got an overlay of scalable vector graphics – everything rippling and folding and looking misty – on Web 2.0 and access to a semantic Web integrated across a huge space of data, you’ll have access to an unbelievable data resource.”

Yeah, Web 3.0 is “looking misty” indeed.  But what, you may ask, is the big deal?  Let’s grant, you say, your argument that the vision of what the semantic web will actually do for us is vague at best.  So what?  Why not just throw some tags in a cloud or whatever the hell it is they’re talking about and see what comes out of it?

Glad you asked, again. Again, in a word: work.  As in, it’s way too much work, on many, many different levels.  Since this is supposed to be a developer’s blog, let’s start from the perspective of Joe. Q. Developer as he approaches the wonderful world of RDF, the principal specification for realizing the semantic web.

RDF provides a standardized way to express facts.  That is, an RDF expression is a statement about a resource, in the form of (one or many) subject-predicate-object expression(s).  So the fact “RDF boils the ocean” can be stated (in RDF-XML format — yes, there are multiple, competing formats) like this:

<urn:concepts:metadata-data-models:RDF> <http://yagni/verbs/boil> “Ocean”

Where “<urn:concepts:metadata-data-models:RDF>” is the subject, “<http://yagni/verbs/boil>” is the predicate, and “Ocean” is the object.  As it turns out, any logically expressible statement of fact can be structured as a subject-predicate-object expression.  Some concepts are very unwieldily to express thus, but that’s not my beef with RDF.  Some concepts are hard to express period, and it’s all well and good if some cognitive tool can make hard things easier, but it’s not necessary.  I don’t think that it’s too much to ask that a new tool doesn’t make easy things hard, though, and that’s what RDF does.

Let’s take a look at some code I had to write recently to read an RDF entry on some book data:

def root = parser.parse(xml)
def product = root[om.Product].find { it.attribute(rdf.about) == "$id" }
def shortID = id - "urn:x-domain:oreilly.com:product:"
def title = product[dc.title].text()
def listPrice = product[om.price][rdf.Description].find { it[dc.spatial].text() == "USA" }[rdf.value].text()
// ha ha, you thought THAT was awesome, wait till you see what we have to do to find the author
// first, find the rdf:resource attribute of the dc:creator node
// next, get the foaf:name of the foaf:Person with the rdf:about attribute = creatorRef
def creatorRef = product[dc.creator][0].attribute(rdf.resource)
def authorName = root[foaf.Person].find { it.attribute(rdf.about) == "$creatorRef" }[foaf.name].text()

And that’s in Groovy, a very expressive and powerful programming language.  All I needed to do was read and parse an XML document (I spared you all that), find a specific book, and pull some simple data about that book, like the author and the list price.  In a half-sane XML document the XPath to get the author node, for instance, would look like “//book[@isbn=’9780596123765′]/author”.  But instead, I’ve got to futz around with attributes in namespaces that reference other attributes of nodes in other namespaces to get a simple string’s worth of data.  It should have been easy, but it wasn’t, and it won’t be the next time I have to do something similar.  With RDF, it’s harder than it has to be to do simple things.

And now, at last, we’re getting to the crux of the problem.  In case after case, it’s just too much work to accomplish my goals.  I now officially have more hoops to jump through, and no benefit for having done so. And that’s after someone burned I don’t know how many months marking up the records in the first place. They used to be half-sane XML.  Now they’re not.  The RDF initiative has been work whose primary yield has been… more work.  For everyone involved.  For now on, until the end of time, or at least until the company comes to its senses.  Yeah, the end of time sounds about right.

Now multiply my frustration by ten bajillion, because that’s what would be involved in marking up even a significant portion of the web.  Now raise that to the power of its own factorial, because that’s what it would take to maintain that markup and ensure that it’s accurate as time goes on.

Because, ultimately, the proponents of the semantic web are basing their vision on two flawed premises: (1) that vague promises of highly accurate, context-sensitive search make all this work worth it, and (2) that real people in the real world are going to do what is necessary for the whole thing not only to work, but to keep working.

Back in the far-off days of 2001, Cory Doctorow predicted that the “meta-utopia” as he put it then “would never happen” because it is “a pipe-dream, founded on self-delusion, nerd hubris and hysterically inflated market opportunities”.  Specifically, Doctorow lists seven reasons why the semantic web will never be what Berners-Lee and his acolytes want it to be:

  1. People lie
  2. People are lazy
  3. People are stupid
  4. Mission Impossible: know thyself
  5. Schemas aren’t neutral
  6. Metrics influence results
  7. There’s more than one way to describe something

And while Doctorow’s examples have some specifics that set one to chuckling (Napster?  Oh, yeah, I remember that…) his reasoning is as strong today as it ever was.  Stronger, even.

Some claim that it’s just a matter of time, that RDF and its ilk will triumph, that people only need to see the benefits of semantic markup and they’ll enthusiastically climb aboard.  I think the opposite will happen: as more people get exposed to the complex and finicky world of semantic data modeling, they’ll throw up their hands in frustration and ask for something simple.  Even if it can’t find them a ride to the footie match.