< Back to Blog

The Reporting Engine - behind the scenes

Product

I've found that a lot of people have a better understanding of our system when I explain how the engine processes a report. (Others are very content with "it works.")

First off, the engine does not use Microsoft Office. You design in Office but the server itself does not use Office. And when in AutoTag you generate a report, that generate process uses the .net reporting engine and does not make use of Office, it is merely a self-contained app being run from AutoTag. So you do not install Office on your server (calling Office to generate reports would be a very slow system and would only run on Windows – that's why we don't do it).

The Java engine is pure Java and so will run on any system that has the Java virtual machine. We have customers using us on various flavors of Linux, AIX, Mac, and virtually every IBM mainframe. The .NET engine is pure .NET – 100% managed code. It does not use Java and does not require a Java virtual machine.

We do this with a single code base so that both engines are the same program and we only have to write code once. How do we do this? For the .NET version we use the Microsoft J# compiler which compiles Java source code into .NET CLR (which is the same thing the C# compiler does). About 2% of the code is specific to each platform and the rest is the same code, compiled twice, by the javac and devenv compilers. The 2% that is different is a set of classes where the public class definition is identical for both platforms, but the code in each method is unique to that platform. We use this to wrap the calls for XML manipulation, bitmap generation, etc.

There are three substantial differences between the Java and .NET reporting engines. And keep in mind that AutoTag uses the .NET engine to show you a sample report even if your server uses the Java engine.

  1. The XML datasources are completely different. The Java engine uses dom4j & jaxen and the .NET engine uses the .NET System.Xml classes in the runtime. These work identically except for some corner cases where the XPath spec is ambiguous. My take on the differences is jaxen generally does what appears to be the most sensible interpretation while .NET does what is the most correct interpretation (but it takes a lot of explanation to see that correctness). But the end result is it rarely matters.
  2. The SQL datasources are also completely different. The Java engine uses JDBC and the .NET engine uses ADO.NET connectors. However, SQL is so mature and so well defined that there is no difference in the result.
  3. We use different charting engines for the Java and .NET versions. Charts will look somewhat different. And some of the more esoteric charting settings are only supported on one or the other engine. Both are very pretty, but they are different. (And the gauges are very different – and not the prettiest with the Java engine.)

Ok, now we know how it is built and installed. Next is how the code generates a report. This is pretty straightforward:

  1. The template is parsed (for the OpenXML formats this uses a SAX parser to walk the elements in each of the template's individual xml files). This builds the template in memory as a Document object. This internal object is format neutral although for every format, some properties in the internal object can only be populated for a specific template type. For example, only an Excel template has the concept of conditional cell formatting and so only for an Excel template will that property be set.
    Note: PPTX output requires so many unique properties that you can only create a PPTX report from a PPTX template.
  2. We then use the visitor pattern to break out the windward tags within the template and perform additional preparation of the template. For example, we mark if any of the document has right-to-left text (Hebrew or Arabic) as that requires additional processing.
    Note: parts 1 & 2 are what occurs in processSetup().
  3. We next use the visitor pattern to apply a datasource to the template. We do this one datasource at a time, the engine evaluating and applying each tag as it hits it in the template. Any tags that are for another datasource are copied across as is. You can apply as many datasources as you wish. And if you apply a datasource that the template has no tags for, it has no effect on the report.
  4. The final step is to lay out the final report. With the previous effort we have the full report as a series of paragraphs and tables, but it needs to be laid out on the page, adding soft line and page breaks and determining the position of every part of the document on the page. This is fully used for PDF, TXT, and printer output where we must draw the report on each page. But parts of it are needed for every output format. For example, if you have a Table of Contents in a Word document, we have to write out the page number of each chapter in the TOC field – Word does not recalculate this when opening a document.
    This part is probably the most complex component of the engine. First off, the rules are complex, very very complex. Second, you have to lay out a page to see if that layout is acceptable. For example, if you have a series of paragraphs marked keep together and keep with next, you need to lay out all of those paragraphs to see if they fit on the present page. If they do not, you have to jump back, insert a soft page break, then start that series of paragraphs on a new page.
    Note: part 4 is what occurs in processComplete().

So there you have it. The concept is simple, but the effort under the covers is significant. I look back at how little we handled in version 1.0 vs. what we do now and I'm amazed at the breadth of properties that must be taken into account and the literally thousands of special cases, rules, etc. we have learned about and added to our code.

But the core API of the server has never changed, and I take a lot of pride that we've kept it simple through 10 versions. And we've kept installing it on a server equally simple – just copy the files onto the server and call them from your application.

Tags Start & End

Tags Can Start & End Anywhere

Appendix B

.NET code for multi-page image output

Appendix A

Java code for multi-page image output

Data Bin Search

The Data Bin can now be searched to find a table, column, node or other piece of data without scrolling through it all.

Shrink to Fit

This will shrink the contents of a cell until it fits the defined cell size.

Time Zone Conversion

A new Windward macro has been added to help with converting dates and times from UTC time to the local time zone.

Image Output Format

New image output formats added.

PostScript Output Format

PostScript, commonly used with printers and printing companies, has been added as an additional output format.

New and Improved Datasets (Designer, Java Engine, .NET Engine)

Datasets have been re-written from scratch to be more powerful and easier to use.

Stored Procedure Wizard (Designer)

This works for all tag types that are connected to a SQL-based data source (Microsoft SQL Server, Oracle, MySQL, or DB2).

Boolean Conditional Wizard (Designer)

Before, conditional statements could only be written manually. Now they can also be built using our intuitive Wizard interface.

Reorganized Ribbon

The ribbon menus have been re-organized and consolidated to improve the report design workflow.

XPath 2.0 as Data Source

Adds various capabilities such as inequalities,descending sort, joins, and other functions.

SQL Select Debugger

SQL Select  Debugger

  • The look and feel was improved
  • Stored Procedure Wizard
  • Improved Exceptions pane

Tag Editor/Tag Selector

Added a Query tab as a field for typing or pasting in a select statement

  • Color Coding of Keywords
  • TypeAhead
  • Evaluate is now "Preview"

Rename a Datasource

All tags using that Data source will be automatically updated with that name.

Connecting to a Data Source

New single interface to replace 2 separate dialog boxes

Tag Tree

Provides a display of all the tags in the template, structures as they are placed in the template. Provides an easy way to see the structure, go to any tag, and see the properties of a tag.

Added Javelin into the RESTful Engine

Support for Google Application Engine Integration

The ability to integrate the Windward Engine into Google’s cloud computing platform for developing and hosting web applications dubbed Google Applications Engine (GAE).

Additional Refinement for HTML Output

  • Improved indentation for ordered and unordered lists
  • Better handling of template header and footer images
  • Better handling for background images and colors

Redesigned PDF Output Support

This new  integration will allow for processing of complex scripts and bi-directional  text such as Arabic.  Your PDF output  will be much tighter and more closely match your template, and we’ll be able  to respond rapidly to PDF requests and fixes.

PowerPoint Support

Includes support for new ForEach and slide break handling, table header row repeat across slide breaks, and native Microsoft support for charts and images.

Tags are Color Coded

Tags are color coded in the template by type, making it easy to visually identify them.

Increased Performance

Version 13’s core code has been reworked and optimized to offer a reduced memory footprint, faster PDF generation and full documentation of supported features and limitations in the specifications for DOCX, XLSX and PPTX.

Advanced Image Properties

Documents can include advanced Word image properties such as shadows, borders, and styles.

Improved HTML Output

Windward has updated HTML output to reflect changing HTML standards.

Version 13 New Data Sources

Windward now works with a slew of new datasources: MongoDB, JSON, Cassandra, OData, Salesforce.com

Generate Code

Designer (fka AutoTag) Generate Code tool allows you to open an existing template and, with a click of a button, automatically create a window with the code needed to run your current template with all data sources and variables in your .NET or Java project. The process is quick, shortens delivery time and helps alleviate any bumps in the road.

Pivot Tables Adjusted in Output

Any pivot tables in the template are carried over to XLSX output. The ranges in the pivot ranges are adjusted to match the generated output. So your final XLSX will be pivot tables set as expected in the generated file.

Imported Template Can be Set to Match the Parent Styles

In an imported sub-template, if its properties for a style (ex. Normal) differ from the parent template's properties for the style, the use in the sub-template can be set to either use the properties in the sub-template, ot the properties in the parent.

Tags can be Placed in Text Boxes

Tags can be placed in text boxes. Including linked text boxes. This gives you the ability to set the text in a textbox from your data.

Tags can be Placed in Shapes & Smart Art

Tags can be placed in shapes & smart art. This gives you the ability to set the text in a shape from your data.

HTML Output Supports Embedded Images

When generating HTML output, the engine can either write bitmaps as distinct files the generate HTML references, or it can embed the images in the HTML providing a single file for the output.

Footnotes & Endnotes can Have Tags

You can place tags in pretty much any part of a template, including in footnotes & endnotes.

Document Locking Supported in DOCX & XLSX

Any parts of a DOCX or XLSX (PowerPoint does not support this) file that are locked in the template, will be locked the same in the output.

Specify Font Substitution

If a font used in the template does not exist on the server generating a report, the font to substitute can be specified.
In addition, if a glyph to be rendered does not exist in the font specified, you can specify the replacement font. This can be set distinctly for European, Bi-Directional, and Far East fonts.

Process Multiple Datasources Simultaneously

When Windward is merging the data into a template, it process the template handling each tag in order, and the tags can pull from different datasources. This allows the select in a tag to use data from other datasources in the select.

David Thielen

President/CEO at Windward Studios

From his early years as a Senior Developer at Microsoft, to legendary designer of the popular Enemy Nations strategy game, to reporting and document generation guru, Dave has never lost his passion for building superb software and teams.

david@windward.nethttps://www.linkedin.com/in/davethielen/

For over 10 years, Windward has lead the industry with our world-class document generation platform that creates visually stunning, data-powered documents designed exactly the way users want and are created in a fraction of the time and cost compared to existing solutions. Proudly located in Boulder, Colorado, Windward Studios is the premier solution for developers and business users adding reporting and document generation capabilities to their applications in over 70 countries around the world.

Contact

Got questions about reporting and document generation? We've got answers—let's connect!
Send a note