31
Oct 11

Developer-oriented features and experiments

Tilt is a Firefox addon that lets you visualize any web page in 3D. A new update is available, coming with more developer-oriented features. Try the addon.

Since the first alpha version of Tilt was announced (a Firefox extension focused on creating a 3D visualization of a webpage), a lot of work has been done to add a great number of developer-oriented features. These focus on debugging the structure of a webpage, inspecting styling and attributes for each node and seamlessly refreshing the visualization when the DOM structure changes or after contents of document are repainted.

Solve nesting problems

Tilt is useful when searching problems in the HTML structure (like finding unclosed DIV elements for example) by providing the extra third dimension, layering each node based on nesting in the DOM tree. Stacks of elements visually represent branches in the DOM, and each node can be inspected for the inner HTML contents, its computed CSS style and the attributes.

Clicking anywhere on the visualization highlights a color-coded rectangle surrounding the corresponding node. Double click shows up the source preview for that node. Tilt also tries to show the most relevant information when needed (one is most likely to inspect the attributes of an input, button or image element, for example, but can easily switch between HTML, CSS and attributes view at any time).

Minidom map

The “minidom” is a tree view representation showing a minimalistic snapshot of the document object model. Each node is assigned a color associated by tag name (blue for div, green for span etc.) and represented as a strip, along with visual markers for the id and/or class if available. Each one of these strips also has a width relative to the type, id and class name length for the respective element, and the corresponding 3D stack in the visualization has color-coded margins. The coloring for individual elements is easily changeable using the color picker near to the minidom legend.

Clicking a strip in the tree view (or directly a stack on the 3D document visualization mesh) also highlights the node with a colored quad. This behavior is a good way to relate with the Style Inspector, and a more unified interaction between Tilt and other Developer Tools is planned in the future. All of these additions make it easier to analyze the bounds of each node, along with the HTML, computed CSS and attributes.

Realtime editing

Because Tilt is able to detect when a webpage’s DOM structure changes or when a repaint is necessary, integration is seamless with existing Developer Tools. Using Tilt and Firebug or Style Editor at the same time is easy. One can enable or disable CSS properties, changing the style of a node, and the visualization changes accordingly.

To enable realtime updates for the 3D webpage, go to the Options menu and check “Refresh visualization”.

Useful for learning

Developer tools such as “view source” have always been used to help people learn about web development. The 3D view highlights the structure of a page better than a flat view, thus anyone can immediately understand the parent-child relationship between nodes in a webpage, their positioning and how the layout is influenced.

One use case for this is the Hackasaurus mashup. The X-Ray Goggles is a nice and fun tool designed to make it easier to learn about the different document node types, the “building blocks” which create a webpage.

Export

A requested feature was the ability to export the visualization as a 3D mesh, to be used in games or other 3D editors. Tilt adds the ability to export to .obj, along with a material .mtl file and a .png texture (a screenshot of the entire webpage). The open .obj format ensures the fact that the mesh can be opened with almost any editor. Here’s a ray-traced rendering of hacks.mozilla.org in Blender:

Fun with experiments

As soon as it was released, many people found clever and interesting alternative ways to interact with Tilt. One experiment was creating a 3D visualization of an image, by exporting chunks of pixels to a HTML representation. The result was a voxel-like representation, with node blocks and stacks instead of pixels. A simple Image2Tilt converter was written in JavaScript, and you can try it directly in the browser.

Accelerometer support was another addition based on community request. This shows how easy it is to add functionality that wasn’t originally planned.

You can view the source code, fork it and also contribute to the addon with ideas or feature requests on Github, at github.com/victorporof/Tilt.

Available as an addon

The latest version of Tilt can be found on Github, but you can also download Tilt as an addon from addons.mozilla.org.

For compatibility, Tilt requires WebGL capabilities. Go to get.webgl.org to check availability and troubleshoot any issues. The current version works with Firefox 6.0 to latest 10.0 Nightly releases (latest Nightly builds now also support WebGL anti-aliasing, working great with Tilt).

To start Tilt, hit Control+Shift+M (or Command+Shift+M if you’re on Mac OS), or go to Web Developer -> Tilt, available in the Firefox application menu (or the Tools menu on Mac OS). You can modify this hotkey (and other properties) from the Options menu after starting Tilt.

More information about Tilt, the development process and milestone updates can be found on blog.mozilla.org/tilt.

Future

Tilt has become an active Developer Tools project, and an ongoing effort is made to integrate it with other existing tools like Style Inspector and Style Editor (source code and latest builds). As the 3D view of a webpage has proven to be useful for debugging, this main functionality will gradually become part of Firefox in future releases.


01
Sep 11

Version 0.82 released

Download: Tilt.xpi
August was a busy month. Tilt development was focused on finishing the user interface and adding an improved user experience. We also added more developer-oriented features and functionality.

The most interesting addition was the “minidom” — a tree view representation in the lower left, showing a minimalistic snapshot of the document object model of a webpage. Each node is assigned a color associated by tag name (blue for div, green for span etc.) and represented as a strip, along with visual markers for it’s id and/or class if available. Each one of these strips also has a width relative to the type, id and class name length for the respective element. The coloring for individual elements is easily changeable using a color picker.

Clicking a strip in the tree view (or directly a node on the 3D document visualization mesh) now also highlights the node with a colored quad. This behavior is a good way to relate with the Style Inspector, and a more unified interaction between them is planned in the future (along with other developer tools like Style Editor).

All of these additions make it easier to analyze the bounds of each node, along with the html, computed css and attributes (which are available in the popup panel editor).

A requested feature was the ability to export the visualization as a 3D mesh, to be used in games or other 3D editors. In version 0.82, Tilt adds the ability to export to .obj, along with a material .mtl file and a .png texture. The .obj open format ensures the fact that the mesh can be opened with almost any editor, from Blender to Maya. Here’s a ray-traced rendering of hacks.mozilla.org in Blender:

The exported .png can also be used to create full screenshots of webpages, as it saves the visible part of a document along with it’s “offscreen” elements.

The three months dedicated to developing Tilt in the Google Summer of Code timeframe have finished, but that doesn’t mean an ending for improving this extension. It’s worth mentioning that version 0.82 of Tilt is now waiting approval to be public in addons.mozilla.org and a great thing to know that, already, even though Tilt isn’t public yet, there are already over 8,000 users.

Implementing more and more features is a sure thing in the near future. The nearest milestone (0.9) will add enhancements like: highlighting and scrolling a selected element in the “minidom”, making a selected strip center/zoom the node stack in the visualization, adding controls to move up/down the dom tree in the editor panel popup, and adding more functionality for the user interface. See all the TODOs and the planned features for more details. You can try some of these additions in the beta from the r90 branch on github.


26
Aug 11

Hackasaurus X-Ray Goggles Mashup

The X-Ray Goggles is a nice and fun inspector tool, designed to make it easier to learn about the different document node types and “building blocks” which create a webpage, letting you “see inside the web”. To quote, “Hackasaurus makes it easy to remix and change any web page like magic”.

This is a nice opportunity for Tilt and Goggles to work together and create an interesting mashup, having some gamification ideas in mind. The first game-like experience will be adding a Portal style transition between two webpages, to symbolize finishing a level and entering a new one.

Since Tilt was originally thought out to be more of a developer tool than a game-oriented extension, and new features designed to better integrate with Goggles don’t always make sense from a developer point of view, a new branch will be created especially for this mashup.

See the Hackasaurus games and blog for more information.


07
Aug 11

Profiling Javascript functions

Tilt has reached a point in which profiling is mandatory. Execution time is virtually impossible to compute correctly in Javascript, but the bottlenecks can still be successfully spotted by counting the function calls and computing the total execution time. Moreover, intercepting function calls can be tricky, but easy once everything is handled properly. If a parent object and the function name are available, using closures we can retain the original function. The whole idea is to overwrite that function to handle before/during/after method calls, and, once this is done, we can record the number of calls, the total time necessary to execute a function and other useful information.

A tricky issue can appear when an overwritten function needs to return a value. In this case, the afterCall still needs to be executed after the function returns. The try/finally clause does just this, it ensures the execution of the afterCall function even after the original function has returned a value.

Using an array of profiles, retaining information about all the functions, variables like longestTime or averageTime can easily be updated. Other issues may appear when handling the “this” object, which can differ from the function parent object. To fix this, the handy .call() hack does the trick.

Probably the most annoying hack needed to be done is passing parameters from the overwritten function to the original child function. Since most of the times the overwritten function has one or more arguments, simply passing the arguments property inside the function isn’t enough. We need to construct the parameters directly, separated by commas, just like a normal call would be executed. Although discouraged, eval() works pretty well in this case, and even with every possible function call profiled in Tilt, no performance drop was observed. Be careful though, and use this hack only with code you trust. Update: a much better way of handling this scenario is to just use method.apply(object, args); to handle passing parameters from one function to another. The code was updated to reflect this change. Never, ever use eval!

Once everything is finished, logging can be done by sorting all the recorded function calls, timing and other information by a key. With Tilt, the most useful data was received when sorting by the total time necessary for a function to be executed. However, this is easily changeable to suit other scenarios.

Because some functions inside objects can be duplicated when creating object via var foo = new MyObject(), that is, when they are declared inside the constructor function and not the object prototype, we need to check for duplicates and recalculate the number of calls, longest time, total time, average time for these situations. Replace the Tilt.Console.log call with your favorite logging method, like console.log or something similar.

Intercepting functions in an object is very easy. Inside the constructor function, call the intercept method:

See the Tilt.Renderer constructor for an example. Certain functions from an object can be specified to be logged, instead of all the child functions. If you’re not using objects, pass window as the parent object for specific named functions, in which case the call would be intercept(null, “window”, “myFunc”). See the intercept function documentation from Tilt.Profiler for more information.


27
Jul 11

Progress

Tilt is getting closer and closer to a beta development stage, let’s see what’s been going on.

The most important added feature is the html preview window, currently using the Cloud9 source code editor Ace. Each time a node is double clicked, a popup panel shows up, displaying the formatted, syntax colored inner html for that particular node, along with the class and id if available. Since it would have been a tremendous work to implement a rich text edit box in WebGL, the easiest hack was to use an iframe containing a simple html document embedding the Ace editor.

This simple page is contained in an iframe defined in xul, inside a panel. On a double click event, all we needed to do was to get the iframe content document, access the <pre> editor element by id, set its inner html to the corresponding inner html of the selected node, then call a function to format and highlight the text. It’s really easy to implement such panels in Firefox, as ‘openPopup()’ and ‘closePopup()’ functions do all the heavy lifting.

The xul responsible with the iframe:

This isn’t, however, all the planned functionality for the html panel. A great and useful feature would be to actually edit and update the webpage rendering in real time as the code changes, and this is planned to be implemented in the following months. This behavior can lead to interesting functionality, and combining it with the Style Editor and Style Inspector would make a nice development/debugging environment. Thus, the next best move is to replace the Ace editor with the Orion Code Editor, actively developed in the devtools area.

To achieve this functionality, an optimized equivalent of canvas.drawWindow is needed, to minimize memory usage and seamlessly update the visualization as the DOM and style changes. A lot of progress has been done with #653656, and the first Javascript shim implementation is available. Once the specifications are consolidated and agreed upon, the MOZ_window_region_texture WebGL extension will be implemented in Tilt.

The way rendering webpages is done has changed quite a bit in the past months. The first experiments were focused on a more translucent visualization and now, all the nodes are rendered opaque. The community feedback has been great in both cases, so the best idea is to keep both rendering methods, and also adding customization functionality. The best way to achieve this is to implement a simple UI with sliders and other elements controlling hue, saturation, brightness, contrast and render mode (wireframe or opaque) for the visualization. Thus, transitioning between multiple ways of visualizing the webpage can be done seamlessly, and fine tuned values can be perfect for different goals.

The current GUI is a work in progress implementation in WebGL, as it would have been too much of a performance hit to implement simple sliders and buttons in XUL. The first mockups for the user interface have been finished:

A couple of other things are planned for the user interface, the most important ones being:

  • zoom and rotation sliders (+ auto rotation)
  • customizable thickness and spacing between nodes
  • preview pane in the lower right (showing the visible area of the webpage, like a map)
  • DOM tree structure in the lower left (along with a search field, go-to-node functionality)
  • reset, zoom in, zoom out buttons
  • menu in the upper right (including the export button)

For now, the major focus is implementing all of the UI functionality, to make Tilt more useful and friendly for debuggers both from the HTML and CSS world. It’s been quite a tremendous effort to create a special UI library and event modules in WebGL, but now that most of the elements have been finished, implementing the actual UI should be the next step.

As a final note, Tilt has been released as an alpha preview last week on hacks.mozilla.org, receiving a lot of feedback from the community. Most of the suggestions will make it into Tilt in the near future, the most interesting ones being:

  • exporting the visualization to different formats (especially 3D models – COLLADA, .OBJ), feature which would add a lot of possibilities like importing into games; the visualization will also be exported to other WebGL supporting browsers
  • modifying and updating the 3D webpage mesh on the fly (as the webpage changes, exposing CSS transforms for each node), meaning that interactive movies, animations and even “DOM art” can be done using Tilt (see this video, creating a 3D Reddit logo).
  • adding a zoom control which isolates a DOM node and it’s children only
  • display useful information on the sides of the nodes too (also a color code to make the most common html elements appear clearly)
  • implement CSS editing functionality, along with the HTML editor panel
  • using head tracking from a webcam to modify the perspective of a webpage

 


13
Jun 11

Visualizing the DOM nodes depth and coordinates

The most important part of Tilt was actually visualizing the DOM nodes depth and coordinates in 3D manner, by positioning them relative to parent offsets and sizing them based on the client width and height dimensions. This is exactly what’s been implemented in the past week, so let’s take a look.

This is an idea of the direction in which Tilt should evolve, as seen on the wiki page:

Tilt mockupFrom the start, the general concept was that “Tilt allows a person to instantly see the relationship between various parts of a web-page and their ancestors in a fun and graphical way”. As you can see, by layering the DOM nodes based on the depth as stacks, positioning and even texturing them accordingly makes a very interesting web-page inspector. These “layers could be built from DIVs, ULs or any containing node with children” and also “elements with absolute position or floats”.

Therefore, two main challenges arise so let’s analyze each problem and find the solution.

  1. Acquiring the texture of the webpage
  2. Traversing the DOM to create the visualization mesh

To get the texture of the webpage, until a WebGL extension will be created to access the actual pixels and share memory with the renderer, a shim could be implemented in Javascript to simulate this behavior. This is easily done using the canvas.drawWindow() method, as explained in the “Drawing graphics with canvas” article on the MDC docs. For security reasons, this API cannot currently be used by Web content. It is chrome only. Long story short, this is the implementation (for more code, see jTiltUtils.js and jTiltExtensions.js on github):

The second challenge was traversing a document object model and building the visualization mesh from it. This is done quite easily using a recursive function and harnessing the Javascript beauty of lambda functions. Starting from the top node, traverse each of the children and call a function:

Quite elegant. Moreover, the get the coordinates (x, y, width and height) for each node, we make use of the offsetParent, offsetTop, offsetLeft and clientWidth & clientHeight properties:

If you want to see more details on how the visualization mesh with each stack shape is build and how the texture coordinates are calculated, check out the jTiltVisualization.js file. Here’s a quick look on how things look for now:

Note that a single mesh is created! Drawing chunks of 3D models, each one with it’s own texture (for example) would be highly inefficient, thus some clever optimizations are done here. An entire object, built of triangles is computed, without duplicating any vertex or corner (by using index buffers), and a single texture is applied to the entire mesh, by computing the u, v coordinates relative to the width and height of the DOM texture.

The following videos are a proof of concept of visualizing the DOM nodes depth and coordinates. The only thing remaining to do now is make the extension usable, ergonomic and fun. But you can check it out now!

Having fun with DOM nodes and effects, and applying the document texture:


08
Jun 11

Tilt at Summer Web 2011

Summer Web 2011 is an event organized in conjunction to the first edition of the Design Jam Iasi, “a two-day venue, during which people team up to discuss, share ideas and solve engaging UX challenges”. The event is non-profit and supported by the Mozilla Labs Concept Series. Find more information on the Summer Web blog.

During this event, I’ve presented and explained the Tilt project to a wide audience of “neophytes, apprentices, and experts in Web technologies and HCI (Human-Computer Interaction)”, and received amazing feedback and interest from the community.

Most of the questions were regarding the schedule and deliverables, but one interesting aspect worth mentioning was about the actual rendering of the webpage itself and not the visualization (which is achieved using WebGL).

The main problem is grabbing the 2d rendering of a webpage, to calculate texture coordinates for each node in the DOM and applying the texture to stacks like in this draft concept of Tilt. Actually, creating a new webpage renderer would be a very bad idea, so Tilt will take full advantage of the implemented Gecko engine, continuously under development at mozilla.org (“the open source browser engine designed to support open Internet standards such as HTML, CSS, the W3C DOM, XML, JavaScript, and others”).

Fetching the webpage texture will be done using a WebGL extension, MOZ_dom_element_texture, built in collaboration with Cedric Vivier from Mozilla DevTools. This is an optimized equivalent of the canvas.drawWindow() function, and it would be used for rendering the DOM elements in WebGL texture. “The extension would be listed by getExtensions from privileged/chrome JavaScript only (similarly to canvas.drawWindow)”.

Take a look at the presentation:


06
Jun 11

First implementation

A lot has happened in the past few days, and interesting progress has been done with Tilt, both from a developer point of view, and from the community side. But first thing’s first: Tilt is an extension, there’s working code on https://github.com/victorporof/Tilt (currently there are 19 followers and counting! lots for such a short period of time), so let’s see what it’s made of.

Project structureThe project structure is quite standard and follows the lines of any Firefox extension, with an install.rdf file containing description about the project (id, name, description, homepage url etc.) and also the target application, which is currently set to a Firefox minimum version of 4.0 and maximum version 7.0a1. I will probably change these as Firefox 7 ships in about five weeks, and it’s a pretty good idea to support even Aurora and Minefield releases. Currently, Tilt is not bootstrapped (it needs a Firefox restart when installed), but I don’t see it as a problem right now. As the project advances, I will most likely use Mozilla’s Addons SDK to achieve this.

The chrome.manifest is composed of three sections: content, locale and skin. MDN defines chrome as: “the set of user interface elements of the application window that are outside of a window’s content area. Toolbars, menu bars, progress bars, and window title bars are all examples of elements that are typically part of the chrome.” These files are all packaged in a JAR archive.

 

From the top, the content contains mostly .xul and .js files. The XUL holds the user interface and (view) and Javascript is responsible for the logic (controller) of the extension. For example, this extension will create an iframe overlaying the current browser parent node (and adding it to the stack), while this is done by accessing a menu under “Tools” -> “Tilt” -> “Initialize Tilt”.

Here is how the XUL currently looks like, also: what happens when initializing Tilt. These two files (browserOverlay.js and browserOverlay.xul) are at the highest level of the extension content. Most of the other files represent the engine to render everything (I called it jTilt for now), some helper draw functions built on top of this engine, and the visualization logic. Other helpful libraries like glMatrix are used, providing great performance, so that everything is efficient to achieve smooth rendering.

An important requirement of any Firefox extension is making it “localizable, basically the locale .dtd and .properties files containing all of the strings, messages and titles or captions of any view component. Thus, it’s easy to translate this extension in many different languages and making it scalable from this point of view.

The third and final part is the skin, with .css files defining the look and other assets (.png, .jpg, images in general) used by the extension.

I’m currently using a TextMate development environment, but there’s also a Komodo Edit project file if you prefer it instead. There’s no “right” IDE for developing Firefox extensions, and any text editor you’re most comfortable with will do. Read this for more information.

Currently, you can find the latest .xpi file in the bin folder. Drag and drop it in your Firefox window to install it. Start it by accessing “Tools” -> “Tilt” -> “Initialize Tilt”, and you should see a rotating quad, tinted with a green color and a texture loaded from chrome:

Tilt screenshot

Tilt screenshot (6 Jun, 2011)

The Bugzilla page for this project is Implement Tilt: a WebGL-based 3D visualization of a Webpage, I’ve received some quite useful feedback already. You might want to cc yourself there to receive updates on the development as soon as they appear. Of course, this blog has RSS, and I encourage you to subscribe to it.

A few other things worth mentioning are some bug pages on Bugzilla:


10
May 11

Tilt: a WebGL-based 3D visualization of a Webpage

Tilt represents a new way of visualizing a web page. This tool creates a 3D representation of the document, with the purpose of displaying, understanding and easily analyzing the DOM. It will take advantage of the great tools Firefox has to offer, as it is an extension which contains a WebGL implementation, providing rich user-experience, fun interaction and useful information, while taking full advantage of 3D hardware acceleration, GLSL shaders and what OpenGL ES 2.0 has to offer.

The implementation consists of a Firefox extension containing a 3D representation of a web page, as both a fun visualization tool and a developer-friendly environment for debugging the document’s structure, contents and nesting of the DOM tree. Various information besides the actual contents will be displayed on request, regarding each node’s type, class, id, and other attributes if available. The rendering will be dynamic, in-browser, using WebGL and GLSL shaders.

This blog will be weekly updated with progress about the development. In the meantime, for more interesting stuff, see

The source code is available for download on github. Check it out!