Let your users mark up images on iOS with Drawsana

We recently released Drawsana, an open source iOS library in Swift that makes it easy to add image markup features to your own iOS app! This post explains why we did it and how it works.

Asana’s iOS app helps you capture thoughts and information at the moment you think to do it, using the device you carry with you. One way the app serves that purpose is by allowing you to attach images to tasks. Sometimes the image itself isn’t enough; you want to call out a specific part of the image. So we built a feature called markup, which lets you add shapes, drawings, and text to images before attaching them to a task or conversation. Here’s how it works:

Of course, you could always do this kind of thing using a separate app like Annotable, and Apple built a native markup feature into iOS for specific situations, but we want to let people capture important information effortlessly right inside the Asana app.

When we started looking at options for implementing this feature, we did a survey of open source libraries that would handle the hard parts for us: efficiently rendering freehand strokes with Core Graphics, object-oriented tool design, undo/redo stacks, and interactive text entry. Given that there are so many illustration apps on the store, we thought it likely that one of them would have published an open source component. In practice, there were no options that fit all of our criteria:

1. Allows us to build a great UI 2. High-quality rendering 3. Maintainable code 4. Either a minimum set of tools (pen, box, arrow, text), or good extensibility

The one that came close, ACEDrawingView, had all the necessary features, but also came with some hard-to-debug issues, inflexible design, and inflexible UI elements that weren’t a fit for our app. We also considered using a web view and a JavaScript library I wrote a few years ago called Literally Canvas, but the complexity of maintaining a bridge and an embedded web page would have been as burdensome as writing a native component in Swift from scratch.

So during our mid-year Roadmap Week, I used my time away from the sprint cycle to write a high quality, extensible library for building iOS markup features that would be useful not only to us, but also to other iOS developers adding similar features to their own apps. It was more work than using an existing library, but the result was exactly what we wanted.

Before I talk about the library itself, I’d like to call out Let’s Build: Freehand Drawing in iOS by Miguel Angel Quinones, which I used as a starting point for learning the basics. It’s a well-written, useful resource.

Design considerations

Besides collaboration apps like Asana, markup can be leveraged for anything where a feature requires communicating an idea outside of the core content of an image. Telegram, a messaging app, lets you mark up images before you send them. File sharing apps like Dropbox could also benefit. It’s silly that all the individual developers at these companies have to go on the same journey of discovery to learn how to build freehand drawing and editable text box features.

Serving such a variety of apps means that extensibility and customization are necessary. It’s not enough to build a decent UI and a framework for defining custom tools. The UI itself is a liability for an open source project that’s trying to fill this niche. Mobile designers at tech companies are rightfully opinionated about feature sets and UI conventions.

So the design goals when choosing the library were distinct from the goals we had when writing a library:

1. Avoid built-in UI 2. Good-looking, performant rendering 3. Useful set of common tools 4. Exemplary Swift code 5. Extensible design

We think we achieved those goals, and we call the result Drawsana.

Using Drawsana

Here’s a short example showing how to use Drawsana to let a user draw a picture and then save it to JSON or a UIImage.

You can find a complete working example on GitHub.

What does it do?

If you’re just trying to ship a markup-like view with as little effort as possible, Drawsana has a nice set of built-in features:

• A simple API that always needs a little bit of configuration, but not too much

• Most of the important tools from MS Paint: pen, eraser, box, ellipse, text, and selection

• Undo/redo

• JSON saving and loading

For those who need more tools outside the built-in set or want to build sophisticated interfaces, Drawsana provides public APIs for everything it can:

• Building tools

• Customizing the text tool’s appearance and behavior

• Adding new kinds of shapes and undo/redo actions

The data for a drawing session is stored in a Drawing object, which has a size and a collection of Shape objects. A Shape is simply something that can be rendered in Core Graphics, and optionally respond to a hitTest(point:) -> Bool method. Changes to the drawing are made using the DrawingOperationStack, which can apply or revert DrawingOperation objects like AddShapeOperation, RemoveShapeOperation, and ChangeTransformOperation. For example, here’s an operation that adds a shape to the Drawing:

Interactivity is handled by DrawingTool objects, which handle touch events and render shapes as they are created by the user. Built-in tools include PenTool, EraserTool, LineTool, RectTool, EllipseTool, ArrowTool, SelectionTool, and TextTool. Here’s an example showing how to implement a tool for drawing lines:

There’s more to it under the hood, and this summary skips cool details like how to implement selection behavior, but these are the core concepts: Drawsana handles the hard parts and lets you hook in where it makes sense for you.

Implementation challenges

Freehand path rendering

When a user drags their finger over the screen, the OS reports it as a series of (point, velocity) tuples. If we naïvely construct a path of line segments connecting those points, we’ll have two visual issues. First, if the user moves their finger quickly or the app drops frames, you’ll see sharp angles in the path. Second, the longer the line gets, the slower it will be to render.

These are the issues discussed in the post linked in the first section, Let’s Build: Freehand Drawing in iOS. Drawsana’s implementation doesn’t differ much from the one described in that post, but it generalizes the strategy for minimizing unnecessary work by managing the set of UIImages that contain permanent and temporary versions of the drawing.

Text editing and rendering

UIKit provides great APIs for measuring and rendering text, but we don’t just want to render text—we want users to edit it in place. That means syncing up a UITextView‘s position and styles with text rendered by Core Graphics. It isn’t a complex concept, but in practice, the tiny details took a lot of tweaking and testing to get right.

Serialization

We didn’t initially have serialization as a feature goal, but when everything else had been implemented, I realized that all the Shape implementations could conform to Codable with little effort. At that point, I thought it would be simple to make the Drawing conform to Codable too. I was so wrong! 80% of Drawing.swift is serialization code.

The complexity comes from two sides of the same problem: the list of shapes is heterogeneous. Each list item conforms to Shape, but that isn’t enough to tell Encodable or Decodable which class implementation to use at runtime.

When I realized the problem, I was honor-bound to find a solution. This library is about solving the hard parts of drawing, not turning a blind eye to difficulty! Fortunately, through a combination of scouring the web, reading documentation, asking for help in Slack, and trying different permutations of Swift code, I solved both problems.

The key to encoding is relatively simple. Rather than trying to encode the list of shapes, I had to create a small wrapper struct. The problem and solution are discussed in detail in Dynamic Encodable with type erasure by Sergey Gavrilyuk.

The solution for decoding is much more verbose. The only solution I could find that worked was to try every shape’s decoder on each array item, and keep the result if it succeeded. This strategy resulted in a minor API wart: if you implement a custom shape, you need to hook into this obscure closure to make decoding work correctly.

What’s next?

We hope that by releasing this library, we help unlock the creativity of app developers everywhere by breaking down barriers to turning ideas into apps. We also hope that if you like it, you help us improve it—Drawsana is MIT licensed and open for contributions! We’re interested in your feedback about how well it works for your needs, as well as any suggestions for API improvements or implementations of new tools and features. Come on over to GitHub and get started!