How I prepare data-merged designs for print using open source software

I co-organize some popular conferences in Amsterdam with a few friends/colleagues. As the resident designer, I’m often tasked with making the name badges that everybody wears. I used to make these in Adobe InDesign, which has a mail merge function that, while kind of weird (I seemed to be required to use UTF-16 CSV files to merge with), worked pretty well.

Since I switched my OS to Ubuntu a while back, I haven’t had a need for InDesign. Really, the only thing I ever used it for was for making badges. That doesn’t justify the cost to me, not only financially, but also in disk space. :-) So I’ve set out to make these badges in one of my favorite drawing tools: Inkscape. Since Inkscape is an SVG drawing program, and SVGs are text, I figured there might be some flexibility there.

That flexibility and freedom have a price, though. Instead of doing everything in one GUI app, a few different programs are involved. The workflow involves using the command line. It’s not a perfect process, but it works fairly quickly and well for me. That said, feel free to laugh at my awkward process. Open your copy of InDesign now. By the time you’ve finished reading this, InDesign should be open and ready to go.

Note that I’m not doing book design here; my use case is fairly simple. If print design were still my profession, I would most likely use the best software for the job.

My biggest problem with this workflow is that I kept forgetting what I did the previous time. That’s the main reason for me writing this, and for a couple of people who requested it. It’s essentially documentation for me, so I don’t forget again. That said, someone else somewhere might find it useful. This is for informational purposes only, and may not work on your system. I am unable to provide any support. It’s probably best not to follow along at all if you’re not somewhat comfortable in the command line.

I mainly work on a Linux machine, and this description is written from that point-of-view. Anyone following along on other systems might have to work out discrepancies or weirdness in their own environment.

With the disclaimer out of the way, let’s look at the process. This might seem a bit complex, but from my perspective the process is simple, and the following things are true:

  1. I draw my artwork in Inkscape.
  2. I have a CSV with attendee data.
  3. I type one command to generate a file for each badge.
  4. I type one command to combine those files into a single PDF.
  5. I send the PDF to the printer.

Note that in a GUI, instead of typing a command, I would probably choose various commands from one or more menus. So, to be clear, from my perpective, this is not more work than doing this all in a GUI program. I will concede to the fact that there is more complexity in the tools I use, as you have to install three, and they might not work the same on every OS. This process assumes a Linux-like OS, and enough knowledge on your part to recognize things you might need to change to get them to work on you particular OS. Instead of InDesign (and a tool to convert a UTF-8 CSV into UTF-16), I need three tools, which I’ll describe below. These tools are freely available and don’t take up much space. The time it takes to generate the badges is not longer (and often shorter) than doing the same with commercial tools.

Badges are made in Inkscape, and data from CSV is merged using a Ruby gem called inkscape merge. It’s not ideal, but it works fairly well in that Inkscape can be used and I’m not locked in to a specific software vendor’s software and workflow. The process outlined here is simple and has kinks, but lots of things could be scripted to do more complex stuff, like customizing visual differences between speaker badges and attendee badges, etc.

1 Things you’ll need

We’re using the following open source tools (please refer to the individual sites for installation instructions):

  • Inkscape, for doing the badge artwork in SVG
  • inkscape merge, a Ruby gem, for doing a “mail merge” of a CSV file with the artwork, which results in separate files for each (side of each) badge
  • Ghostscript, to merge all the files you generated into a single PDF, if you’ve exported those files themselves as PDFs. If you’re on OS X or Linux, there’s a good chance this is already on your machine.
  • ImageMagick (specifically `convert`), to merge all the files you generated into a single PDF, if you’ve exported those files as bitmaps, e.g. PNG.

If you’re using OS X or Linux, there’s a good chance Ghostscript is already on your machine. The last three tools are command line tools, and even Inkscape can be run from the command line (and in fact, inkscape_merge does exactly that). Don’t be afraid to try it out, if you’re so inclined. That’s how one gets more comfortable with the command line.

2 Creating the Inkscape and CSV files

Badges are made in Inkscape any way you want. You’ll have to have a separate file for each side of the badge, unless both sides will be identical.

I draw my own crop marks for the printer in my document, put them in their own Inkscape layer, and lock it. Then I draw the rest. You can also do this after the artwork, but I like to get it out of the way at the beginning. If I want to show my colleagues what a badge will look like, I’ll make a selection of the printed area and use Inkscape’s “Export bitmap” function to make a PNG of it to show. Obviously, I’ve already consulted with the printer about what they need to receive from me before I even start. I’ve worked with printers for whom InDesign was literally the only option. We like our current printer.

Conference name badge in Inkscape GUI
A layout in Inkscape with variables

Text that is to be replaced by CSV data should be indicated like so: %VAR_csvColumnName%, where csvColumnName corresponds to a column in the CSV. The CSV might look like this:

name,company,twitter,
Stephen,Zero Interface,@stephenhay,
PPK,Quirksmode,@ppk,
Krijn,Qontent,@krijnhoetmer,
Martijn,,@Martijnvduuren,

This is essentially the following structure:

name company twitter
Stephen Zero Interface @stephenhay
PPK Quirksmode @ppk
Krijn Qontent @krijnhoetmer
Martijn   @Martijnvduuren

If I were to have a bit of text in my Inkscape SVG file with %VAR_name%, then when I run inkscape_merge with the above CSV file, that variable would be replaced with “Stephen”, “PPK”, “Krijn”, and “Martijn”, each in their own file.

The CSV file should be UTF-8 encoded.

2.1 A caveat about the CSV file

Some weird bug I’ve been getting using inkscape_merge prevents me from changing the column separator to something other than a comma, which is the default. This is a pain, because sometimes you’ll have a field in the CSV that uses commas, like “ACME, Inc.”.

I don’t know enough Ruby to figure it out. For future reference, this is an example of the type of error I get:

inkscape_merge --csv_col_sep='|' -f badge.svg -d data.csv -o batch/badge_%d.pdf
/var/lib/gems/1.9.1/gems/inkscape_merge-0.1.3/bin/inkscape_merge:33:in `block (2 levels) in <top (required)>': undefined method `csv_options' for #<OptionParser:0x00000002322088> (NoMethodError)
        from /usr/lib/ruby/1.9.1/optparse.rb:1360:in `call'
        from /usr/lib/ruby/1.9.1/optparse.rb:1360:in `block in parse_in_order'
        from /usr/lib/ruby/1.9.1/optparse.rb:1347:in `catch'
        from /usr/lib/ruby/1.9.1/optparse.rb:1347:in `parse_in_order'
        from /usr/lib/ruby/1.9.1/optparse.rb:1341:in `order!'
        from /usr/lib/ruby/1.9.1/optparse.rb:1432:in `permute!'
        from /usr/lib/ruby/1.9.1/optparse.rb:1453:in `parse!'
        from /var/lib/gems/1.9.1/gems/inkscape_merge-0.1.3/bin/inkscape_merge:11:in `<top (required)>'
        from /usr/local/bin/inkscape_merge:23:in `load'
        from /usr/local/bin/inkscape_merge:23:in `<main>'

The only workaround I have right now, since I currently don’t encounter many fields containing commas, is to create separate SVGs with hand-entered data for each row in the CSV that uses in-field commas, and convert these separately. It’s an awful workaround, but my last set of badges only had two instances, so no one dies.

BTW, no, using double-quotes in the CSV doesn’t work, as inkscape_merge seems to skip rows containing them. Joy.

3 Running inkscape_merge

First, make sure there’s a directory to put the batch input. inkscape_merge is going to create one file for each row in the CSV. This might be a lot. Let’s say that directory’s name is batch.

inkscape_merge is then run as follows:

inkscape_merge -f sourcefile.svg -d datafile.csv -o batch/badge_%d.pdf

Where sourcefile and datafile are the names of your SVG artwork and CSV file, respectively. The %d will be replaced, for each file, by the number of the CSV row.

This will put as many files in the batch directory as there are rows in your CSV. Remember that rows containing in-field commas should simply be exported by hand from handmade SVG copies of the artwork in which you’ve entered the actual data of that row instead of %VAR_whatever%. You can do that from the command line:

inkscape --without-gui --export-pdf=outputfile.pdf --export-dpi=300 sourcefile.svg

where sourcefile is the specific SVG file and outputfile is the name of the file you want to end up with.

A little trick I use is to leave double-quotes around fields containing commas in the CSV. Since inkscape_merge doesn’t output these, I know which ones I have to do manually, and I also know how I should name them according to row, and these filenames are missing from the batch directory since they haven’t been generated.

3.1 A caveat about exporting to PDF

Inkscape allows you to do lots of stuff with SVGs, and these things don’t always export well to PDF. It’s hard to say exactly what, but sometimes I notice problems with transparency and masking/clipping. If you run a batch to PDF and open one and see that it’s not as you intended, you can export all your rows to PNG instead. Since the DPI is set to 300, this should suffice for most print work:

inkscape_merge -f sourcefile.svg --format png -d datafile.csv -o batch/badge_%d.png

This will give you a bunch of PNG files instead of PDF. You’ll also have to export your infield-comma exceptions to PNG manually.

OS window containing 4 files
A separate file is created for each row in the CSV file.

4 Combining the batch files into a single PDF

Printers generally want PDFs, so whether our batch consists of PDFs or PNGs, we’re combining them into a PDF.

Make sure you’re in the batch directory first.

4.1 Combining multiple PDFs into a single PDF

To do this with PDFs, we use ghostscript:

gs -sDEVICE=pdfwrite \
    -dNOPAUSE -dBATCH -dSAFER \
    -sOutputFile=combined.pdf \
    `ls`

For more info and variations, see https://www.linuxjournal.com/content/tech-tip-using-ghostscript-convert-and-combine-files.

The `ls` turns the output of the ls command into input for the gs command. You could also use * or $(ls) instead, in this case. If you’re in the batch directory, the output of ls is a list of all the files that we generated before.

4.2 Combining multiple PNGs into a single PDF

For PNGs, we’ll use ImageMagick’s convert command:

convert -limit memory 2MB `ls` combined.pdf

We’re limiting memory usage to 2MB here, because otherwise with many files, sometimes the system will kill the process because it’s using too much memory.

Window of a PDF viewer program showing 4 pages
The separate files are now combined into a single PDF file

Finally…

That’s it! This is how we’ve been making most of our conferences badges for over a year now (except for one which involved HTML, CSS and PHP). The above process is pretty painless once you’ve run through it a few times, and generating a file with about 300 badges only takes a few minutes. Since SVGs are text, and inkscape_merge variables can replace any text in the SVG, you can customize colors per badge or practically anything else. You could potentially do any kind of scripting you like. And of course, you could wrap all of this in a tidy bash script for easy execution.

I’m in the process of making a demo video of this workflow, in which I also demonstrate using variables for colors. This should be done in a day or two, and I’ll post a link here when it’s done!

Reality is messy

We like to believe that the right framework, method, approach, or conventions will allow us to create perfect projects. Our code will be beautiful, pure, performant, and maintainable. Outside of some developer blogs however, this is often not the case.

Reality is messy.

Some people are fortunate enough not to have to deal with this reality on a daily basis. There are people who work for browser-makers. Their projects are not most projects. There are people who work for large-scale and popular “startups”, who have the luxury of focusing on a limited number of components. Their projects are not most projects. Although we can (and do) learn a lot from posts on Medium about month-long experiments in link underlining, many of us have pressing deadlines. Processes can be messy because they involve other people and limited resources, tight budgets, difficult stakeholders and ridiculous time constraints.

Outside of the product world, many of us work in creative and/or technical services. Our job is to help our clients. These clients have concerns during the design and development process that we need to address. Sometimes, we can’t do everything exactly as we would like. Client needs trump our own.

I’ll give you an example: a project I worked on where my job was to help take an existing desktop-centric design and make it responsive. My task was to determine exactly how everything would work responsively. My deliverable was a series of more than 20 design comps built with HTML and CSS. The comps were themselves responsive, so stakeholders and the developers knew what the end result should be.

Here’s the thing: there were extreme time constraints. We basically had two months to get this fairly complex website responsive. In order to do this, the developer would change as little HTML as possible, as this would mean rewriting components all over the place. Yes, we wanted to see that happen, but it wasn’t feasible. So I used the existing HTML whenever possible, and I cringed while doing it, because holy shit, this HTML (with all due respect to the developers; you know CMSes). But it was practical and allowed the developer to reuse a lot of my CSS. Which was a lot more CSS than I would have had to write if I were able to rewrite the HTML.

Could I have said no? Sure, but I wouldn’t have been helping anyone, least of all myself, by doing so. This client has interesting challenges, and I wasn’t willing to just say no to four months of work with them because of a tight deadline.

Another issue I had to deal with was working in weekly (!) sprints. Coming up with responsive strategies for each component of certain screen types and then prototyping them in one week was not easy. In order to do it without creating conflicts and still get things done on time, I had to namespace each logical group of components, like this:


.product {
  /* Styles, including media queries, here. */
}

.account {
  /* Same idea, different logical group */
}

While doing this for each round of logical prototype groups allowed me to avoid conflicts with prototypes I had done or those I would do in future sprints, it made me repeat myself across those blocks, since I had literally no time to constantly refactor. And it was not my job to create the front-end code, though I knew the developer was using and altering snippets from my CSS to save time. Result: my comps had way too much CSS.

Could I have an approach like BEM? Yes, I could have. But since some CSS methodologies eschew inheritance, I lose a huge time benefit. And remember, HTML changes were required to be minimal, and that’s not feasible when choosing a CSS approach that involves moving the dirty work into class attributes.

What’s your goal?

In the project above, the thing that kept me sane was focusing on the specific reason I was asked to join the project: to decide how everything should look and work responsively, and to visualize this for stakeholders and the developers, in such a way that they could actually experience it on actual devices so that it is clear what needs to be done.

That had to be my focus.

We finished the work in two months with just a few people. And a phase two was planned to go in and refactor, to improve lots of things, including performance. That’s the right time for optimizing HTML and CSS, and improving things like performance. Ideal? Certainly not. Realistic based on the needs of the client? Absolutely. This particular client sees their website (100% of their revenue comes from the website) as a process rather than a product. I think that’s a healthy way of looking at things. It means they have a responsive site, built within two months time, that customers could enjoy right away. That site, even with sub-optimal HTML and CSS, performed twice as well as the previous desktop version. Customers don’t care about what the source looks like, and phase two, almost underway, is intended for code optimization and other quality improvements.

Messy, isn’t it? It certainly is. But really, it’s often the reality of a service business. It’s natural to try and find processes or approaches that work for every project, but you won’t. And they don’t. Embracing this fact is one step toward pleasing clients, and also helps deal with the frustration that we can’t always make everything as perfect as we’d like it to be. Of course, that shouldn’t keep us from trying.

I start every project with a lofty goal of achieving perfection. I’ve never reached it. I make mistakes all the time. I look back and immediately see things I would have done differently. I try to learn from those mistakes. I try to remember that making those mistakes and dealing with the messy reality of the work, while still trying to do my best work, puts me on a path of constant improvement.

We need to give ourselves a break. I hope I’m learning more and becoming just slightly better every day. I hope you are, too. In our line of work, where everything moves so quickly, that’s something to be happy about.

Here’s to a better imperfect year than the last one.

Choosing between min-width and max-width media queries

I’m often asked a variation of the following question:

Should I use min-width or max-width media queries?

The obvious answer is, of course, “yes”. But you know what they’re asking: which of the two is better?

Those for whom responsive design has become second nature might find it an odd question, and will know that the answer is, “it depends”. But having seen lots of responsive design implementations, it’s clear to me that many designers and developers aren’t quite sure. And I’m happy to share my own opinion of what “it depends” means regarding this particular question.

Desktop first and max-width

There’s a decent number of designers/developers that still think of “the desktop” as their primary design manifestation, with other (generally smaller) sizes as secondary; often these appear as afterthoughts, since there can be significant visual clues as to the amount of effort that has gone into these other screen sizes when compared to their desktop counterparts. Often, when one examines the CSS of these sites, `max-width` is the media feature of choice.

This makes sense. If a design is built desktop-first, then all the CSS for the desktop-ish version of the design has been written, and must be overridden with more CSS for any other breakpoints. If a given desktop-ish width is our baseline and we are unwilling or unable to refactor our CSS, then it seems logical to override what is now our base CSS for all viewport widths with those we only want to apply to smaller widths.

This can lead to a situation I’ve often joked about: writing styles only to undo them later. This is an example I often use (assume .related takes the form of a sidebar):


.content {
  width: 60%;
}

.related {
  width: 40%;
}

@media screen and (max-width: 37.4em) {
  .content,
  .related {
    width: 100%;
  }
}

The above example omits a lot, but I’m sure you get the idea. This approach, when used over a bunch of components, can greatly increase the amount of CSS needed to complete the project. But the main problem is how illogical it is, because when embracing the fact the block-level elements are 100% of their parent by default, then the following makes much more sense:


@media screen and (min-width: 37.5em) {
  .content {
    width: 60%;
  }

  .related {
    width: 40%;
  }
}

Here, we’re making use of the default state of block-level elements, and overriding them when they need to change from the default. Again, it might seem clear to some readers, but it doesn’t take looking at the source of many websites before you understand why I’m clarifying it here.

Using max-width against your better judgement

There are a couple of reasons why, even when you know better, you might deliberately choose to use an approach similar to max-width approach above:

  1. You’re a developer and you received desktop-only, desktop-first, or other-sizes-as-an-afterthought comps from the designer. How can you know? If you’ve only designed or received designs for desktop-ish widths, this is what you’re dealing with. If you’ve got nice detailed desktop-ish designs with a few “mobile” or “tablet” examples thrown in there, this is what you’re dealing with. In these cases, may the max-width be with you, unless you have a budget and/or schedule that allows you to refactor during development.
  2. You’re making an existing website responsive, and the existing CSS is design baggage you’ve inherited and cannot change, for whatever reason.
  3. You’re compensating for the fact that we don’t currently have element queries, and you’re handling this with CSS instead of using a JavaScript polyfill for a non-existent spec.

So, what’s the best to use?

Here’s my clarification of “it depends” for this particular question. And remember it’s just my own opinion based on my own experiences: look at default display of a given element. If you have to override this default for smaller screens, use max-width. If the default is usable on small screens, use min-width, and only when the element needs to deviate from the default. And of course, I recommend letting the content determine where that happens.

A good example of using max-width to override an element’s default display so that it works better on smaller screens is when using tables. Imagine a table that contains too much content to display in its default form on small screens. One approach might be:


@media only screen and (max-width: 30em) {
  .big-table tr,
  .big-table td {
    display: block;
  }
}

This turns each row (and cell) into a block on narrow screens. The table can become quite long vertically and more styling is usually required, but it’s often better than the back-and-forth horizontal scrolling that would otherwise be required to read the content.

In this case, it makes all the sense in the world to leave the table in default form, except for in browsers that understand the media query and where the screen is no larger than, in this case, 30em.

Other elements (I would venture to say most other elements) that have defaults that work fine on smaller screens, only need changing when necessary on larger screens; using min-width is then a solid choice.

In short: let element defaults help determine which media feature to use in your media queries.

Thanks to Brad Frost for reviewing this. He also has a post with related topics.