Smart Filters–i.e. those that can be adjusted or removed at any time, leaving the underlying pixels unaffected–address what is probably the single longest-standing feature request in Photoshop. Customers’ response to them has been quite good, but the details of how & why they work as they do may be a little subtle. For example,
- Why can’t you paint directly onto a surface that has a Smart Filter applied?
- Why are you limited to having one filter mask per layer (instead of having one per filter)?
- Why do Smart Filters add file size?
If you’re interested in the story of how and why Smart Filters came to be as they are, read on. I find the whole topic of how Photoshop is evolving from a simple "a pixel is a pixel" app into a dramatically more powerful editing pipeline fascinating, but I recognize it’s not everyone’s cup of tea. 🙂
Around four years ago (while building CS1), we took a crack at enabling non-destructive filtering. After talking with quite a few customers, we opted to enable "Filter Layers"–just like adjustment layers, except containing filters. We quickly encountered a number of problems, both technical and conceptual:
- Performance: It’s one thing to run something well on a screen-resolution document, and quite another to run it well on the huge multi-layer images that Photoshop is often asked to handle. Yes, computers keep getting faster, but cameras and other devices generate ever more data, and our collective patience keeps getting shorter. There were too many times when performance was lousy–say, whey trying to paint layer 20 out of 50 with a bunch of filter layers applied. We therefore required filter layers to turn off while editing the layers to which they were applied, after which they would be re-applied. This made for a pretty lame user experience.
- File Size: There’s no free lunch: you can either pay as you go (storing both "before" pixels and "after" pixels in a document), or you can pay later (not storing the pixels in the doc, and requiring the filters to be re-applied when opening the doc). If you don’t store the filtered "after" pixels, then the PSD file can’t be opened by apps that don’t have the same filters installed (bad for backward compatibility, cross-app integration, etc.). We opted to store the after pixels, but because adjustment layers are very small, people were then surprised that something that looked similar would add file size.
- Editability: This may be the toughest of the three, because it’s conceptual, not technical. If you apply a transformation that substantially changes the appearance of the underlying pixels (e.g. Polar Coordinates is one extreme; see screenshot), what should happen when you attempt to paint or otherwise edit those pixels? As you drag a straight line to the right, what should you be seeing through the prism of the filter(s)?
So, with knowledge in hand but not much else, we pulled the plug on filter layers. As work progressed on CS1, we thought about ways to build a solution that not only addressed the pain points above, but that solved other Photoshop limitations (scaling and warping non-destructively, integrating with Illustrator and other apps, etc.). That’s when Smart Objects were born.
Okay, about the name: I technically work in marketing, but I hate cheesy names, and making everything "live" this or "smart" that starts to have a whiff of fromage to it. Finding a simple, factual name proved difficult for a feature that’s really general–one that enables, among other things:
- Turning one or more layers into something that can be scaled, rotated, or warped losslessly
- Creating multiple instances, all of which update when one is edited
- Preserving complex vector data & re-editing it in Illustrator
- Preserving raw data & re-editing it in Camera Raw
- Potentially enabling linking to external files
- Potentially lots more that I can’t get into here/now
So, we went with a pretty generic name–"Smart Objects." The concept is identical to the Smart Objects feature that’s been in GoLive for several versions: place something the app doesn’t understand natively, and on the fly turn it into a native version, while keeping the source & output linked together. People sometimes get hung up on the one implementation difference (GL links whereas PS embeds), but doing so misses the main point: on-the-fly conversion from {whatever data} to native pixels.
As the basis for Smart Filters, Smart Objects address the three pain points of Filter Layers listed above:
- For performance, they don’t set the expectation that you can be drawing/painting/moving things while watching the results update on the fly.
- For file size, they create a visible, new entity, and they require a conscious choice to create it. And it’s the same hit you take for enabling re-editable scaling and warping.
- For editability, again they don’t set the expectation that you can paint while seeing the results applied. You simply edit the un-filtered/-warped/-scaled pixels, hit save, and then see the transformed results.
That’s why you can’t paint directly onto a filtered surface, and why you can have only one filter mask per Smart Object (because multiple masks would lead you to expect that filters would update while painting on masks underneath them).
This entry is pretty long already, so I’ll cut myself off, but in the next day or two I plan to post an entry that discusses some of the challenges involved in making Smart Objects easier to use.
Thanks, John, for explaining this so clearly.
[Glad to hear it’s useful, Klaus. As you’d imagine, I’ve been waiting to talk about it for a long time. –J.]
I’ve lusted after “filter layers” since forever, and have been repeatedly disappointed with various PS releases when this never happened — until now. After having played with the beta, it’s clear to me that the Smart Filters will probably be my No. 1 reason to upgrade to Cs3. So it’s reassuring to know you guys have been sweating over this stuff — also since forever! 🙂
Thanks for the info, I’m always curious to learn more about how the program ticks. Every now & then I even stumble onto the kind of math that’s involved.
Oh so this must be what you were talking about in the previous post: Secret Life of the Smart Filters.
Thanks John. This is totally my cup of tea. And being on the outside of things, I often wondered why non-destructive layers took so long to make their way into Photoshop. I also found it curious that it became an extension off of Smart Objects. But after reading all this, it makes total sense.
I also used to wonder why applications such as After Effects and Illustrator could offer non-destructive filters and Photoshop couldn’t. But now I get that it had a lot to do with the medium and managing expectations. After Effects users are already expecting to have to wait for a preview to render and Illustrator you’re mainly dealing with vectors. So that argument doesn’t really hold up to the what Photoshop users expect and build. Remember the old 99 layers limit? I do.
Thanks again. Good stuff.
Thank you John. I just blogged about that very issue: http://www.90percentofeverything.com/2006/12/25/photoshop-cs3-usability-is-more-than-just-ui/
My hunch was a little bit off. I assumed that there was an architectural reason why it was technically infeasible whereas the decision was made on grounds of user experience. But it’s a shame you didn’t allow it as an option. Maybe you could have popped up one of your quaintly old fashioned modal dialog warnings on first usage? 😉
Many people aren’t working on huge images and it smacks somewhat of collective punishment to withhold functionality on the grounds that some people would find it a performance hit.
[There’s certainly room for improvement–lots and lots, we know. –J.]
There is also the point I made in my post which is that you have made Photoshop more complex conceptually as a result of this and for reasons that are not intuitively obvious to the user.
[Ah–wait ’til you see my next post. –J.]]
This entry was indeed a very interesting post. Really enjoyed reading this behind-the-curtain information 🙂
Though I’m not using filters that much, it surely adds great value to PS CS3 and once again convinces me to upgrade, maybe..
Smart filters were the BIG lack of Ps for centuries. While any video post-production software could do it since the 1st version, why Ps was unable to do it?
There’s no technical problem on it. Just show me the result, but don’t render it. Non-destructive filters are essential on any post-pro software.
Maybe Adobe could look at other softwares, and observe how easy and effiently they ressolve this question.
[I feel like you’ve missed what I wrote about performance and direct editability.]
Mr. Adobe: have a look at Shake, and learn on how to manipulate pixels and colors.
[Does Shake let you paint directly onto filtered pixels? I don’t know of any apps that do, but I’d be happy to look at some examples. –J.]
(Plus: smart filters don’t visualize correctly: aply a simple blur on a smart layer and get crazy looking how the blur stops at the edge of a virtual box.. until you render it.)
[I’m guessing that’s a bug. –J.]
thanks for details. i like how hard you’re pushing the blog with extensive in-side information and open communication.. keep it!
[Cool–thanks. –J.]
regarding ‘smart’ filters- i can’t get a particular reason why filters and objects got ‘smart’ while color adjustments are done with ‘adjustment’ layers. e.g. imho filters and color adjustments are actually more or less similar pixel level edits by it’s nature.
[The difference is that adjustments (as PS uses the term) affect just one pixel at a time (no dependence on surrounding pixels), whereas filters affect more than one pixel at a time. That’s why filters take longer to run, and why Shadow/Highlight has never been available as an adjustment layer. –J.]
so- just a theoretial thought: why smart filters aren’t just expanded set of adjustment layers? while color calculations are being done in realtime (i think you aren’t caching adjustment layer results?) and it’s not a speed problem, adjustment layers using more complex filters could probably store “before” and “after” pixels.
for increased efficiency- why not consider an option in preferences how photoshop would store layer’s cache- e.g. saving “before” pixels’ data into separate files and using them only when needed? this would make photoshop document like a ‘project’ with linked files but ram usage would be much lower on the final edit file..
also- i wouldn’t mind if increased flexibility like multiple masks, etc. would require to recompute filter’s stack. e.g. to change something from first filter you’d disable later filters, change what’s need and once reenabled- the stack would re-render further modifications..
i’m writing all this because i think it should be a bit confusing for new users to get the point why ‘photo filter’ can be applied as adjustment layer while ‘stylize>solarize’ is smart filter 🙂
also- if adjustment layers would be combined with smart filters into some kind of unified non-destructive “smart layer” feature- we’d get additional flexibility like being able to insert color adjustments between smart filters.
[I think we’ll be able to make improvements in the future, and it’s good to get the info out here so that we can get ideas and feedback. –J.]
..once again i finish in general concept where every operation is represented like some kind of node in process tree as used in compositing applications (shake, fusion, etc)
p.s. another question but really annoying in my workflow: what’s the problem with being able to LINK mask to smart object?? it’s so cumbersome limition – if i can move smart object and separately the mask- why i can’t relatively move them both on the same time??
[I know; we’ll see what we can do to address that. The technical details behind the limitation are more than I have time to get into right now. –J.]
John, on a related note: Ever since Smart Objects arrived, it’s annoyed me greatly that the files are embedded — and not linked externally. The same word, “Place,” is used by both InDesign, Illustrator and Photoshop — but conceptually, they are radically different, due to the embed/link difference. I LOVE being able to externally edit files with PS or Illustrator and have them auto-updated inside InDesign. And I HATE not having this work the same way in PS — and the file-bloat which embedding entails. Could you please comment on this issue — either quickly here or in a future blog?
[Sure, good idea, although the answer is pretty short: we just haven’t had time to do it yet. Linking carries with it a chunk of overhead (UI for link mgmt., error checking, etc.). Anyhow, as far as the concept goes, I don’t see linking vs. embedding as a big difference–more of an implementation detail. Ideally Photoshop can and will offer the flexibility to do either/both. –J.]
(Whatever you do, let me say that I’m very happy with your blogging and all your other tireless work for all us Adobe users!)
[Thanks! After an eye-watering onslaught of icon-related comments, I was getting a little worn down, so I appreciate the encouragement. 🙂 –J.]
The Smart Filters are great! This, more than UI “productivity” is where your energies pay off. I am surprised, however, that you admit to being in Marketing…
[Believe me, no one is more surprised than I! ;-P I didn’t know that product management falls under the M-word title until a week after I’d move across the country to join Adobe & heard an engineering manager refer to “you marketing guys.” Afterwards I turned to another PM and said, “Heh–he thinks we’re in *marketing*!” Er, um, yeah. Anyway, my dissatisfaction with this large company’s occasional inability to spit out info cleanly & quickly is a big reason for the blog. –J.]
I have to admit that the second thing I looked into after trying the smart object filters for the first time was adding individual masks and when I saw that I couldn’t, I sort of thought “Hmm… Well that would have been really useful” and ever since, I’ve been wondering why this seemingly obvious (to the user) feature wasn’t there.
I figured that it probably came down to hardware overhead somehow but I’m glad you stepped in and took the mystery out of it so I can go back to enjoying my three quarters of a glass full of water instead of gazing at the small empty part.
[ 🙂 –J.]
Thanks John! That’s some really interesting and useful info on Smart Filters. Keep up the good work!
Useful info John, it all bewilders me a bit as a non computer educated person who started with a mac 15 years ago! But your blog certainly helps in the undestanding of how it all works.
CS3 seems a quantum leap – thanks to all of the team, LR is great too!!
All the best for 007 !
[Thanks, Geoff; same to you. –J.]
Thanks, John, for such a clear explanation of why things work as they do instead of as we might have preferred.
Out of curiosity, I just fired up Shake and tried what artofbit suggested. I loaded in a 50MB Tiff, put a 50px Blur and then an Edge Detect filter on it, and then went back to do some clone stamping (using a Quickpaint node) UNDERNEATH the filter nodes.
I have to say, I was quite impressed, but it wouldn’t satisfy most Photoshoppers (including myself, I suspect). After an initial lag of some 10 seconds (on a dual 2.3 G5, 4.5GB RAM) I could clone stamp at any zoom level and the update was smooth and live. I assume the lag indicated that Shake was caching the filtered layer to provide feedback. Switching the Blur node off triggered another 10 second pause as the Edge Detect was recalculated, and another similar pause when I resumed stamping.
So it works, but the performance hit is appreciable, as described.
[Thanks for this info, Ben. –J.]
Well I think smart filters is a great idea but the limitation of one mask per smart object is a major limitation. Layer masks are one of the most powerful and important aspects of color adjustments and creative filtering and without the ability to have a mask for each filter in a smart object, the entire useability and effectivenss of this featutre is compromised. I urge the design team to work out a way to add layer masks to each filter in a smart object.
interesting, but many people seem to forget that Live Picture had “smart” filters over a decade ago (only blur and USM, but none the less, it was ahead of it’s time)
[Sure, non-destructive filtering per se isn’t new at all. After Effects has had it since the dawn of time, and Illustrator and Fireworks have been running PS filters non-destructively for many years. The question wasn’t whether it was possible; rather, the trick was to find a UI model that would make filtering flexible and usable across a range of document types, file sizes, etc. –J.]
Also: Smart Objects change Photoshop’s compositing model from linear to branched (where each smart object is a branch from the linear layer model). And since Smart Objects can contain other Smart Objects…. (yeah, I need to work on a UI to show the full graph)
And we really wanted to avoid the “preview now, wait hours to render later” scheme of LivePicture and many of the video/animation applications.
As for what Smart Objects are capable of: you ain’t seen nothin yet!
I for one think the invention of smart objects is a fascinating study model that needs to be applied globally to Photoshop and ingrained as a core functionality of the application at the root level. This is the only way we print professionals will ever get some kind of hold of the mess that is currently the norm for print. Unfortunately, due to the lack of real time editing of SO’s, it makes the feature almost useless to solve this hair ball of an issue.
What needs to change?
The ability to edit SO’s directly in the parent layer, yet retain the original untouched SO in child form. The interaction of root layers is KEY to the success of this paradigm in print process control at the file creation stage. The architecture of Photoshop has to be thought in terms of two implementations. Editing and Color adjustment and color space conversions. As of now, the methodology of the development is a free for all – chaotic blend of whatever the user can think of. In order for ORDER to be acquired, the Adobe Photoshop team must embrace process control at the file stage and not expect a user to understand or want to understand what color management is and why it’s there. Smart Objects somewhat address the dire need to make color management seamless or transparent to the general user which is god sent, but SO’s are lacking in implementation for real file manipulation in real world situations. A Photoshop document needs to be one big Smart Document whereas – a child document within a SO is where the mess an artist or photographer can play.
If I had a deeper understanding of the limitations of SO’s and the challenges that the engineers have with respect to said topic, I could give better guidance in what needs to transpire to solve the pandoras box of file output.
Great, GREAT read, thanks!
I just found this site, I’m immediately adding it to my “must visit constantly”!
Ted Dillard
Smart Object Evangelist
Being able to link in external files have been my biggest PS-want for years. Please add it soon!