RubyTML
A TML framework for Ruby

Coalfire has completed their code review and has signed off on the PCI-DSS readiness of the RubyTML framework, clearing the way for our next major release: we’ve been hard at work on RubyTML v2.0, which applies everything we’ve learned in the process of working on RTML v1.x and then some. RTML 2 is now officially available, and I’d like to take a moment to mention a few highlights of the project.

First of all, we have some decent documentation this time! It’s not quite finished, but we expect to be wrapping it up by the end of this week, and I’ll be typing up an ongoing tutorial series in the form of blog posts shortly after that. It’s all available (or will be) at http://www.rubytml.com.

This is a complete rewrite of the TML generation framework from the ground up, in which we’ve built up all of the strengths of its predecessor and left the weaknesses behind. It’s faster, more robust, easier to test, and smarter than ever.

Speaking of tests, that’s practically a topic all of its own. RTML2’s test suite presently covers 92% of the entire framework and we’ll be bringing that closer to total coverage (our goal is 110-115%) over the next few weeks; and this test suite is actually capable of binding itself to your Rails application environment so that you can make sure that your code doesn’t break any part of RTML. In addition, we’ve got complete TML support for 3 major test frameworks: RSpec, Cucumber, and the Rails default, Test::Unit. Perhaps the most interesting aspect of testing in RTML2 is that we’ve also got a baked-in TML Simulator that loads the raw, generated TML; parses it; and then processes it just like a terminal would. This allows you to write automated tests using one of the above frameworks that verify that your TML application is behaving as expected – and that opens the door to Behavior Driven Development in TML!

RTML2 is the most extensible and modular version to date. Widgets can now be packaged in standalone RubyGems, allowing you to share them via the RubyGems repository and turn them on or off on a per-application basis. The Widget class itself has been thoroughly revamped, and now rests at the very core of the framework. It has become the single most powerful component of RTML, making possible complex mutations to the Document Object Model (DOM) in a totally reusable and DRY manner.

The new release also makes a reality something that we’ve been dreaming of for a long time: RTML documents can now (relatively) easily be saved to and loaded from the database, opening the doorway to a persistent DOM that we can now manipulate in ways previously unthinkable.

One of the most significant changes in RTML 2.0 is the installation process. Way back in RTML v1.2, we introduced the framework as a Ruby Gem. Back then, all it really did was generate a Rails project and insert the RTML Rails Plugin. Well, we’ve moved away from Rails Plugins completely, now, and moved the entirety of the RTML framework into a Ruby Gem. This makes dependency setup and version control a breeze, not to mention making possible the release of standalone Widgets.

We haven’t lost sight of what made RTML 1.x great, however. All of this is not only available – we’ve also found significant ways to improve upon it:

• Layouts provide a clean, application-wide look and feel
• Model, View, Controller architecture cleanly keeps TML logic out of Rails controllers and views
• Generators automate preparatory code, substantially boosting productivity by creating complete application skeletons within seconds
• Event processing has been streamlined into a single, reusable syntax, providing a more cohesive and intuitive interpretation of the code
• Terminal-specific data is gathered and made available automatically
• A single controller can serve a variety of formats, including TML, HTML, JSON, XML, etc. – with virtually no additional coding

With the increase in modularity, we’ve had a chance to rethink exactly what gets distributed in the core RTML framework, and that process has led us to remove Beryl, the baseline TML payment application. Our mentality here is that, for those users who don’t intend to make use of Beryl, it’s just adding bloat – and that’s never a good thing. That said, what of the people (including ourselves) who do make use of Beryl? For those people, the Beryl application will be released as a Ruby Gem called “rtml-beryl” when it is ready.

Bear in mind that RTML is an Agile project. Release 2.0 is a major milestone, but is not the be-all, end-all of TML generation. There were a few things we wanted to implement for 2.0, but we decided in the end that the release should represent what we recognize as the minimum set of functionality to get the code out the door, and start gathering feedback! That said, here’s a list of what’s still under active development, slated for RTML 2.1:

• One Line EMV Processing – This was a Widget released circa RTML 1.1, and instantly became one of the defining factors of RTML. We’ve not left it behind, but we want to make sure we get it right. EMV Processing was a substantial chunk of code, and it will be again.
• UML Diagrams – These were released with RTML 1.2, and immediately became an essential development tool. They’re still regarded as an essential tool – we just ran out of time. Next release, we promise.
• Web-based Drag-and-Drop Development Tool – This is the elusive GUI that we still haven’t gotten around to naming. Don’t let the lack of a name fool you, though – it’s about 65% complete and we don’t see any reason we can’t have it ready for at least an alpha release with RTML 2.1 (though we’re seriously considering mirroring the Beryl route with this one, and releasing it as a standalone gem).

Naturally, as project requirements shift, we’ll alter our objectives to match. That’s how Agile rolls, after all.

‘Til next time!

Colin MacKenzie IV

One major limitation of RubyTML v1.x is in how it deals with TML variables. While it was still many steps ahead of its pure-TML equivalent, making use of the variables has always felt decidedly unintuitive. They are still remote, client-side variables, and in the end I was never quite able to stop treating a variable as a sort of intangible “thing”. In my apps, at least, TML variables never quite felt like they were storing any data. Assigning a value to a TML variable in its present implementation feels way too much like metaprogramming: writing a program that writes a program. Granted, that’s what RTML is all about, but it’s supposed to make it easy—which, in my own humble opinion, the current variable implementation utterly fails at. Since TML variables can’t really be used until the application is on the terminal, any interaction with them has to be treated basically as static chunks of XML at best, and as flat Strings at worst.

Well, no more. RTML 2.0 is almost here, and with it comes the unleashing of a fully armed and operational Widget. This complete rewrite of the core Widget class makes possible what I’d like to refer to as “High Level Processing”—that is, the ability to treat a TML variable essentially the same as any other server-side variable without losing the subtle reminders that it is, in fact, a remote variable.

Code time! Here’s an example of the current RTML implementation:

# See if we should prompt for cashback. If so, go to :ask_cashback.
doc.screen :check_cb, :confirm_sale do
  next_screen :ask_cashback, :if => 'tmlvar:prompt_for_cashback == 1'
end

# Prompt for cashback. There are hyperlinks in the display that lead to either
# :confirm_sale (if cashback is declined) or :check_cb_amt (if accepted).
doc.screen :ask_cashback do
  set 'payment.amount_other' => 0
  display
end

# Validate that the requested cashback amount is no more than the sale amount.
doc.screen :check_cb_amt, :conf_debit do
  set :message => "Cashback cannot exceed sale amount"
  next_screen :enter_cashbk, :if => "tmlvar:payment.amount_other > tmlvar:payment.amount"
end

# Prompt user to enter a cashback amount, and then go to :check_cb_amt for validation.
doc.screen :enter_cashbk, :check_cb_amt do
  display
end

# Finally, confirm the total amount with the user.
doc.screen :conf_debit do
  set :total_amount => 'tmlvar:payment.amount + tmlvar:payment.amount_other'
  display
end

Now, here’s what the same logic looks like in RTML v2.0:

# Check for whether we should prompt for cashback. If so, display the prompt.
# The :ask_for_cashback prompt contains hyperlinks to either
# :validate_cashback_amount if accepted, or :confirm_sale if not.
# If not, then confirm the sale.
doc.screen :check_for_cashback do
  _if tvars[:prompt_for_cashback] == 1
     # We can also use tvars[:payment][:amount_other] = 0 if that's more convenient.
     tvars['payment.amount_other'] = 0
     display :ask_for_cashback
  _else
     goto :confirm_sale
  _end
end

# Validate the cashback amount. If it's under sale amount, go to :confirm_debit_amount;
# otherwise re-prompt the user, showing an error message this time.
doc.screen :validate_cashback_amount, :confirm_debit_amount do
  _if tvars['payment.amount_other'] > tvars['payment.amount']
     tvars[:message] = "Cashback cannot exceed sale amount"
     display :enter_cashback_amount
  _end
end

# Enter the cashback amount, and then validate it.
doc.screen :enter_cashback_amount, :validate_cashback_amount do
  display
end

# Finally, confirm the total amount with the user.
doc.screen :confirm_debit_amount do
  tvars[:total_amount] = tvars['payment.amount'] + tvars['payment.amount_other']
  display
end

Now, if you have a keen eye, you’ll notice that the RTML v2.0 code is just a few lines shorter (excluding the comments). But that would be missing the point. Readability is the key, here. I don’t know about you, but personally, I find it much easier to make sense of the second code snippet than of its predecessor. The other thing to notice, a little subtler this time, is that we are actually embedding two different “steps” within the initial :check_for_cashback screen. First, we’re checking whether we should prompt for cashback; and second, we’re actually doing it. RTML is going to generate a screen or two for us in the background, totally automatically. You can imagine how, for really complicated pieces of logic, this can begin to generate reams and reams of (necessary) TML with only a few lines of RTML.

And all along, that underscore at the beginning of each condition is a subtle reminder: “Hey, just keep in mind that you’re still working with remote variables.” It’s repeating what you already know, just to make sure you know it: even though it’s easier to deal with remote variables, they’re still remote, and while something like:

tvars[:user_id] = @user.id

might just work, you shouldn’t push your luck with something like…

@user = User.find(tvars[:user_id]) # => Oops, error!

And, finally, even though it’s slightly off-topic, notice the names of the screens in RTML 2. They’re longer and more descriptive. Yeah, TML still only supports a 12-character screen ID, but we’ve worked through that on the RTML side. And yeah, the generated TML is still quite readable. But that’s another post for another day.

Despite my apparent silence, I’ve actually been hard at work on RubyTML 2.0. In this new version of RTML, the Widget plays an even more pivotal role: the entire framework rests on its shoulders, so it’s imperative that it all comes together just right. While writing the new and improved Widget code, I found myself with a strange dilemma. The proxying system that is in charge of adding the Widget entry points (formerly called Accessors) uses modules to mix-in the entry points with its targets. It accomplishes this by maintaining a single Module instance for each target, such as “document”, which then includes the proxy modules from individual Widgets. Think of it as a many-to-many relationship for modules.

Herein lies my problem. When a module is included into a Ruby class, the class apparently creates a link to that module. When new methods are added to that module, the class knows to send calls to those methods into that module. So far, so good. In our case, we’re including a module that has included a module, so it’s got to go a step further. When Class C includes Module M1 and Module M1 includes Module M2, Ruby is smart enough to create these links to both M1 and M2. However, when we then add more methods to M2, Class C can’t detect the change because M1 hasn’t actually changed: it still just includes a module, M2. Technically, no new methods have been added to M1, so Class C can’t pick up on that.

To update the class, then, the module must be explicitly reincluded so that new links can be created. This code demonstrates the issue:

m1 = Module.new
m2 = Module.new
class C; end
C.send(:include, m1)
m1.send(:include, m2)
m2.class_eval { def hello; puts 'Hi there!' end }
C.new.hello          #=> NoMethodError!
C.send(:include, m1)
C.new.hello          #=> "Hi there!"

Since Widgets are now included by symbol, it was not possible to track them explicitly. Luckily, Ruby is awesome so it doesn’t matter. We can iterate through objects of a certain type using ObjectSpace, and we can see if a Class includes a particular module using #include?, so we can do something like this:

class Module # Actually, in RTML we subclass Module to avoid contaminating the environment
  def reverse_include(m)
    include m
    ObjectSpace.each_object(Class) do |o|
      o.send(:include, self) if o.include? self
    end
  end
end

Such a simple, yet effective solution. Still, I can’t take credit. That goes to kch, who had a very similar problem and was able to shed some light on this trick.

-cm4

RTML 1.2 is almost ready. A little behind schedule, this time, which I’m blaming on other projects. The new release will focus mostly on automated testing for TML apps, with particular emphasis on isolating potentially unwanted variable values and screenpaths. One notable feature not related to unit testing that will be immediately useful is the new default page.

Currently, if you run the following command:

—you’ll see that it generates an RTML action and RTML views, but no HTML. Generally speaking, an RTML application won’t cater to a standard Web browser, and so that HTML file is sort of unnecessary.

But if you’re a new user, you don’t necessarily know that—and besides, it just seems like such a waste of a perfectly good action.

So RTML 1.2 will be introducing an additional HTML file for each action generated by the above command. This won’t be retroactive, and will apply only to new controllers. This template figures out whether RTML is running in development, test or production mode, and alters its content to suit accordingly. If it’s run in production mode, the visitor will receive a friendly “go away” message indicating that s/he isn’t meant to be here.

However, if it’s run in any other mode (development or test, for example), it’ll assume the “visitor” is actually an RTML developer, and display a useful cheat sheet to help them get started. The cheat sheet contains information such as file locations, RTML form helpers (and examples), a list of variable operators for assignment or testing equality, a link to the RTML Beginner’s Guide, and, of course, the UML generator.

The HTML template looks a little something like this:

Even before this feature was complete, I found myself referring to it repeatedly when building RTML applications. The UML diagram has become, for me, one of the single most useful features, and has made developing a brand-new RTML app considerably easier because I can visually see my changes just by refreshing the cheat sheet. I now keep the cheat sheet open in one window and the raw TML output open in another, and

Obviously, you can alter this HTML file to provide a more suitable front-end for anyone who might hit your controller by accident, or use it to allow your application to cater to standard Web browsers in addition to TML-enabled ones.

Even though RTML 1.2 itself isn’t ready yet, this piece is complete. If you’re interested, check it out by grabbing a copy of the “edge” branch by running the following from your app’s “vendor/plugins” directory:

git clone git://github.com/sinisterchipmunk/rtml.git
cd rtml
git checkout --track origin/edge

The current RTML implementation will completely replace one screen with another if you call the “screen” method. For the purposes of inheritance, this can be useful, but what if you only want to alter a small portion of a screen (for instance, adding a hotkey to point to the embedded menu)? In this article we discuss the “screens” method (notice it’s plural this time), and how it produces two totally different results depending on how you call it.

If you develop TML in the Canadian or European (or most non-US) markets, you’re familiar with EMV processing. You’ve probably copied and pasted that monstrosity from one app to another since you started writing TML. Well, if you’ve moved on to the RTML framework, copy and paste no more!

That’s because the recently-released RTML v1.0 fully supports EMV processing, and has all the necessary code built into a Widget that can be called with a single line of code. Yes, you heard right: Those 300, 400, 500 or more lines of TML that you’ve been dragging around with you have been completely replaced by just one line. Want to see it?

The overarching goal for RTML is and always has been simplicity. Its design lives up to this standard fairly well, but there’s still one nagging issue to be resolved: the source code itself. See, my vision for a RubyTML Utopia culminates with an app that is capable of completely removing the last vestiges of TML code from the application server. It is one in which the author need only attach non-TML elements, such as payment processing or Web browser front-ends, completely ignoring how to interact with terminals. It becomes just “one of those things” that the Rails/RTML framework takes care of for you.

There’s no doubt about it, we’re a long way from that goal today. But with every passing day, that Utopian ideal comes one step closer to fruition. Initially with the BTMLPA app, and now with the Beryl app, we are seeing design and implementation of TML become progressively simpler and quicker to accomplish. Features lend themselves well to reuse, so that most apps simply need to say “I want a Sale,” and a Layout to decide how it should look on the front-end. This is all great, and it’s slated for initial release with RTML 1.1, making it something that will be available pretty rapidly.

But even as I’m building the list of goals for 1.2, my mind is already wandering to RubyTML 2.0 and beyond. Having tested a new piece of functionality yesterday, I can now say that it is a 100% certainty that there will be full support for database level application development. This means, put simply, that the entire RTML application resides within the database.

Oh, yes I did.

I have a working prototype that stores the “skeleton” of the TML application in the Rails database. It does not store raw TML, which is bulky and repetitive. It does not store Ruby source code, which would require all manner of big, un-DRY parsers to deserialize. It stores a skeleton of how to build the application without actually building it. Sort of like writing a tutorial for the computer. “First create a ‘new document’. Then set the default screen to blah.” Et cetera, et cetera. This means that the database code is future-proof. We can alter any number of internal RTML workings, add methods, do pretty much whatever we want (within reason) to the RTML core, and it’ll still work. Because it’s not storing source code. It’s storing how to use the source code.

So why does this matter, and how does it play into the “simpler world” scenario? Well, think about it. If you have an application that extends some currently-imaginary “RtmlDatabaseController” then all you would ever have to worry about – at least, in terms of controller code, helpers and the like – is the actual Rails source code to accomplish an action. The terminal code is out of sight, out of mind. Unit testing becomes an even tastier piece of cake than it already is, because you can model your entire document using standard Rails Fixtures. You could build the entire document from the console if you wished (though I can’t think why you’d want to).

Oh, but it gets so much sweeter.

Currently, you need to map your /index.tml to something. Now, theoretically, you could play with the routing tables for a few hours to point /index to the index action and .tml to the rtml format. That should work, but only for a single controller, because all of your terminals need a single entry point. If you have two different apps to run, you need to add a controller that becomes nothing more than a glorified router – branching one set of terminals into application A and the other set into application B. That means you’re maintaining a minimum of three controllers (plus views), and who knows how you’ll approach alternative devices like Web browsers or BlackBerries. RTML may make it easy to design interfaces into these alternative devices, but you still need to figure out a way to get the user into the RTML controllers.

Database-level TML logic solves this issue entirely. Controllers in RTML 2.0 will come complete with switching mechanisms to pull the appropriate application out of the database to display to a terminal. This puts the entire RTML interface – controllers, actions, views, everything TML-related – out of sight, allowing you to concentrate solely on your Rails app without worrying yourself with how it’s going to interface with a terminal.

Now, the billion-dollar question: how do you actually build these TML applications for storage? Luckily, the billion-dollar question comes complete with a billion-dollar answer.

By the end of September, I intend to have a prototype GUI to go along with my prototype Dbstore. It’s going to be 100% drag-and-drop, point-and-click, fill-in-the-blank goodness. It’ll run within your Web browser directly from the RTML Core, meaning that your database model will be created in realtime, as you click on the buttons. What you see will be the exact result that the terminal will see. This will include complete testing options, so that you can prefill any necessary TML variables or let RTML do it for you. It’ll include the TML validation that is already a part of the RTML 1.0 unit testing core. It’ll even include a nifty “Activate!” button that will instantly deploy your application for immediate use, with options to throw it into “test mode” so that you can connect your terminal to it without risk of exposing it to the public.

RubyTML has almost reached the point I envisioned it might one day reach. Only thing, it’s gotten there much, much faster than I could have ever hoped. I love Ruby.

Can’t. Freaking. Wait.

In the process of trying to get the rest of Rails to do its thing, I somehow managed to screw up the server configuration. All of a sudden, I couldn’t start Rails. It was complaining that the Rails 2.3.3 gem was missing. What’s more, I couldn’t use any gem.

Testing an RTML document is a bit of a gray area, because this is a region that simply hasn’t been given a whole lot of attention in the past. Thus far, your testing options have been pretty limited: load the document in a browser and see if it breaks, and then if it doesn’t, throw it into an Incendo simulator and see if it dies a hideous death.

Well, that’s all starting to change.

I finally got around to giving the RTML form builders some serious TLC. As I’ve been warning in the Rdoc for ages, I’ve rewritten them completely. They’re not quite finished, but they’re getting pretty close. This is a major milestone for RTML, because it marks the last major feature addition before a 1.0 release. Now it’s just bug hunting. That’s not to say that RTML is complete, of course; but 1.0 is, nearly.

Using a Web browser to preview your code is helpful for several reasons. You have a bigger screen area, so it’s easier to read Rails error messages. You can scan through the generated TML to verify that it’s valid markup. If the Incendo gateway is complaining about an error on a particular line, you can just view the source of the document in your browser and jump to that line to see what’s up. However, there’s always been one glaring limitation of using the Web browser: the faking of terminal data.

So I’ve been battling a particularly mind-boggling issue in RTML lately, but I think I’ve finally found a decent solution.

One of Ruby's great strengths is the ability to modify the very definition of an object on-the-fly. I'm talking about what has most recently become known as metaprogramming, or programming functionality into an object at runtime that you may or may not know anything about. Often, you don't even know what class an object is, and just as often, it's a singleton of Object that's had properties added or changed, but doesn't actually exist as any specific class at all! Of course, this idea of creating objects with no type (though technically its type is still Object) is not new to the world of programming. Python can do the same. Ruby's strength comes not from the mere ability to accomplish these actions, but rather from the incredible simplicity it brings to metaprogramming.

It took awhile, but RTML officially has a home on the Web. No more of this emailing of zip files or checking out of private SVN repositories -- that sort of thing was so mid-90s. Things are going to be different from now on. The RTML source can be downloaded from Github, and issues can be tracked via Lighthouseapp. Online documentation will be hosted here, as well as a forum and, of course, the batch of articles you're reading now.

To understand what RTML is, we have to backtrack a bit and talk first about what TML is.