In 2009 Treyarch pulled me into their studio from Activision's Studio Central division to work exclusively on Call of Duty Black Ops 1 and 2. Implementing these new tools and technologies saved me and others countless hours of grunt work, but also opened up the floodgates tor a whole new world of shader development and demand.

 

I worked on R&D and tools during off season, and during production I worked a double duty of producing a slew of complex shaders, continued development and support for AssetWorks, ShaderWorks, managed and extended the COD shader graph system, the plug-and-play filter system, the PIMP material processor and supporting the rest of the guys who were creating shaders and using my tools. Each system I developed required a great deal of continuted support, and at Treyarch, if you write it, you usually own it, so I defientely had my hands full.

 

Here is some of the key work I did...

 

 

 

ShaderWorks ( shader authoring tool )

 

I continued development of my node-based shader authoring tool and fully integrated ShaderWorks into Treyarch's pipeline which. ShaderWorks greatly simplified the custom shader authoring and iteration process and allowed other developers and artist to play a bigger role in the shader development process. The addition of a visual authoring tool and plug-and-play system was a massive improvement over their old system and allowed engineers and technical artists to work together and collaborte on the creation of complex shaders at a fraction of the time.

 

 

In the end, utilizing ShaderWorks saved myself and others countless hours of pure grunt work and helped us innovate many new special effects, but at the same time, it opened up Pandora's Box and we were constantly flooded with requests by production and other departments. Once you give them a cool shader in an hour, there's no saying no, and understandably, they started treating shaders like regular art assets. The problem was, some shaders took minutes to develop, test and check in, while others required new algorithms be developed, tons of prototyping and iteration, optimizing, dealing with compiler and cross platform issues, engine restrictions and adding missing engine functionality. So keeping up with demand was rough, but it helped us take our graphics output to another level. Before leaving Activision we had 20-25 engineers and artists using the tool in production.

 

Also, to give you an idea of the quantity of code we were dealing with, my high level graph system of 30,000 lines of template/shader code, was used to generate almost 600 shaders totaling upwards of 1 million lines of HLSL/GLSL code. Try managing that without an automated code-generator like ShaderWorks! If they continue to use the tool for next-generation development, these numbers will increase exponentially, but remain manageable.

 

 

 

AssetWorks ( asset management tool )

 

I developed a central asset management application from the ground up to replace Treyarch's old asset tool. AssetWorks was far quicker, very user-friendly, had over 40 new useful features and allowed us to greatly scale the studio's asset collection for Black Ops 2. It is one of their most important tools and is being used studio-wide on a daily basis.

 

 

 

 

P.I.M.P. ( high speed material processor )

 

I rewrote Treyarch's existing material processing system ( which was up to 1000x faster in some cases ) and scalable for production's growing demand for specialty materials/shaders. The system it replaced was nearly impossible to debug and so slow that adding or changing a parameter control to a material would prompt a full convert, which with the amount of assets, was creeping up to 12 hours. The worst part was waiting on the converter for hours to see if you made any errors in the template you were touching so iteration on materials was essentially dead.

 

Developing the material processor, with little knowledge of their pipeline, was one of the toughest things I've done, especially porting over all the touchy template scripts to C++ code was a huge undertaking and to this day I'm suprised it all worked out in the end. PIMP was a big reason we were able to scale up the amount of materials and variations we had. It also opened the door for me to integrate ShaderWorks which gave us a powerful plug-and-play shader system.

 

 

 

Shaders

 

Despite being primarily a software / tools developer, one of my strongest skills and obsessions is shader development. And given that I created the plug-and-play shader ecosystem, I was able to crank out shaders extremelly quickly, and was easilly Treyarch's biggest shader contributor with over a couple hundred used in both Black Op games. If you ask anyone at Treyarch, they would definetely tell you I was the "shader guy".

 

The shaders ranged from a complex GPU-based lens flare system, tension mapping, realistic fullscreen visor raindrops, damage shader for things like the SAM Turret scene, multi-pass X-Ray shader, the spider cam, a very involved subtitle shader for the opening of levels, sonar gun post-fx, script controled multi layer parallax advertisement shader used extensively, special anisotropic lit fur shader for the horses, helped with the rage shader, the main overhead satelite strike force post f/x shader, the warmap shader, the involved holograph table shader, helped with the speciality gun camo shader, emblem and gun camo support in our phong shaders, gradual skin bubbling shader for burn victoms, cool visor shader with breath and cracked screen for the jetpack jump scene, the hologram dancer shader, wall water, the parallax wrist control shader, menu distortion, and the list goes on.

 

 

Click here to expand the list of shaders I developed

• Script controllable multi-layer parallax advertisement shaders, player aware, multi-layer, fully controled by artists and scripters.
• Extensive GPU lens flare system, tied into the particle system ( 100% edited by the FX team using ShaderWorks and the extensive flare system I developed, huge win ) Major issues trying to get the code to play nice with the PC/360/PS3/WiiU compilers, not enough registers, instruction count, flow control etc.. worth the pain though )
• Very complex per-pixel sub-title shader ( huge pain in my ass )
• A slew of simple to complex variations of our skin/cloth shaders for things like burning skin, toxic bubbling skin
• LOD-based tension mapping for wrinkles and cut scenes
• Special projected target player overlays
• A bunch of tricked out gun reticles for things a wide selection of sonar and infrared style guns
• Highly customizable gun/cloth camo shaders
• A ridiculous amount of custom single pass post-fx ( jetpack jump, water hitting the visor for squirrel suit jump, visor mist, cracks etc. )
• A bunch of EMP and other damage post f/x
• Dynamic visor frost for base jump
• High quality ice and snow effects
• Avalanche snow
• Underwater particles
• Visor rain shader, water drops splatting on your visor and leaking down ( all done in a vertex shader, using hundreds of dummy quads )
• multi-pass xray shaders
• Spider cam effect
• All kinds of lighting model overrides ( for example a custom anisotropic horse hair shader or specialized rim lighting )
• A slew of phong and cloth variants
• Blood splatter
• Very off the wall rage post fx shader for a machete masacre
• Extensive multi-pass sonar shader ( looked super cool, but the level was canned )
• All kinds of bink playing post fx shaders
• An insanely involved SR71 cockpit shader for controlling troups using an offscreen render cam
• Shaders for a scene where character is drugged up ( distortions, floating numbers, etc )
• Nitrogen exploding, skin freezing up shader
• The main strike force top down satellite view shader ( needed major iteration because production never knows what they want before asking for it )
• A very involved war room map shader that was controlled via the client script. Building this functionality into a shader and client scripts alleviated a team of UI guys from having to do it the hard way
• A very involved multi-pass war room holodeck table shader
• Various frosted windows ( not used for game play, scattered reads killed the 360 )
• See through bullet holes in walls using multi-layered parallax trick to give the illusion of depth
• Special running water for fountains
• A bunch of animated or scrolling unlit effects for things like signs or walk lights etc, used everywhere
• Cool new gun camo shaders for perks, like the diamond gun shader. I used some tricks for the sparkles, but had some sampling issues across consoles )
• Holograph stripper shader ( was a pain because the guy dealing with engine side render targets had like 4 code paths for the consoles and nothing worked the same. These are the types of things that NEED proper plug-and-play support, because it literally stops you in your tracks )
• Bunch of quad cam tech
• The shader fro the 3D parallax wrist control device
• HUD distortion effects
• Fake fire above water when player is under water

 

 

 

SuperFlare System

 

The studio and FX team had been asking our engineers for years to develop a proper lens flare system. Given the amount of work inivolved of implementing it into the converter, linker and engine, as well as developing a stand-alone editor, it kept getting pushed to the side.

 

With ShaderWorks integrated into the pipeline, I was able to develop an extensive GPU-based lens flare system using only shaders, which would encode all the code variations and parameters within the shader and basically bypass the need to touch the pipeline and engine. The system was a huge success and was used throughout Black Ops 2 ( over 80 flares ).

 

These shaders were pretty complex and I ultimately spent alot of time fighting with the PS3, XBox, PC and WiiU to get around weird compiler bugs, register limitations and instruction count limits. Before leaving, I ported the system over to use geometry shaders for the next-gen consoles, but would have been fun to see where I could have taken this system for their upcoming game...

 

 

 

Tension Mapping

 

I Integrated a new rendering technique into Treyarch's engine/pipeline for realistic dynamic normal-mapped facial wrinkles. It basically dyncamilly computes edge/face tension while a character is animating so that a shader can dynamically blend between a base normal map and two pre-generated compression/stretching wrinkle maps. We mostly used it in single-player and during cinematic scenes due to performance and memory limitations, but it was enough to score myself and two animators a nomination at the 2010 BAFTA Awards for Artistic Achievement. Since then, Activision's Studio Central has been putting a lot of time and effort into developing even more complex facial animation systems for upcoming platforms.

 

 

 

 

Generic Filters

 

I implemented a much needed plug-and-play filter system for quickly adding post-fx and overlay filters into the game without needing to touch the converter/linker or engine. This system was easily setup and controlled via client script, the shaders were effortlessly created in ShaderWorks and tweaked in real-time from AssetWorks.

 

ShaderWorks 2

 

ShaderWorks 1 was a good starting attempt at node-based shader authoring. When I first arrived at Activision I was paired up with one of their most veteran graphics engineer to help re-engineer ShaderWorks more for their game development needs. He implemented the core graph system, template language and code generation, I developed everything else ( application, rendering engine, user interface development, etc ). ShaderWorks 2 was a huge learning experience for both of us. We both had our own assumptions as to what the ultimate node-based shader authoring tool should be and we did our best to come up with something that our studios could use to become more efficient. Looking back, we overengineered some aspects and neglected others. We had a hard time convincing some of our graphics engineers to work with a system like this because it added so much unwanted complexity. I didn't blame them, the template language was using a make shift macro expansion language which didn't have any error reporting and wouldn't fail if errors were made. This made it virtually impossible to debug. The graph system was not very straight forward, and engineers expect to be able to understand everything that's going on under the hood. Also, the generated code was completely unreadible, everything was inlined, and every graph node would create a bunch of redundant variable indirection which made it worst.

 

Despite all this, we used ShaderWorks 2 to ship a handful of games and allowed us to created our first plug-and-play shader pipeline which was a step in the right direction in terms of being able to quickly generate many custom shaders while collaborating with artists. The fact that we were now using a template-based system was a major improvement over version 1.0, which allowed us to re-use code, make fixes and changes and batch-export those changes to our final shaders.

 

 

ShaderWorks 3

 

Since the code generation and graph-system core were still out of my control, I had to continue concentrating on the application itself. I was sick of dealing with buggy third party UI controls and decided it was time to engineer my own user interface API for ShaderWorks since much of the application required custom controls anyway. I ended up using a mix of GDI and GDI+ for the drawing layer and built the API and messaging system on top of MFC. I wrote my own text editor control from scratch, a new featureful graph control, and over 40 other custom controls and UI framework. The API looked and ran great, but lots of optimization was needed to get the GDI/GDI+ UI running well. With this release, I was able to add all kinds of bells and whistles. I had support for multiple workspaces, a 360/PS3/PC viewer that connected live to ShaderWorks and could be launched onto other user's computers, target previews, extended toolbox controls, perforce integration, sub-graphs, etc. I was also lucky enough to hire my good buddy Jamie Parent who I grew up with and worked with at various companies over the years. He helped out on many fronts, from creating an auto-update system, file change notification system, integrated IM system for user support, various custom controls like the file and directory controls, as well as our own custom file dialogs.

 

The main issues left with ShaderWorks 3 were by far the horrible macro expanding template language, unreadable code generation and non-engineer friendly graph system. After many disagreements about the direction of the software, I decided to let our veteran engineer have ShaderWorks 2, and I would shelve version 3.0 while I reworked the core issues which were in my mind holding us back from getting acceptance from the graphics engineers, who are the gatekeepers of our gaming pipelines.

 

 

ShaderWorks 4

 

At this point, I was eager to make ShaderWorks a success and put 100% of my efforts into making version 4.0 as coder friendly as humanly possible by replacing its core graph system, code generator and template language.

 

ShaderWorks 4 was aesthetically the same as version 3 on the outside, but this time around, my main goal was to completely gut out the core graph system, code generator and horrible macro template language. I managed to replace the macro language with a very easy to use C++ style script interpreter, which had full error reporting. I added visual studios style intellisense and code tips. I reworked the code generation to output extremely well formatted and function based shader code so that engineers could actually read and optimize shaders after they were exported. I added very important features like the ability to jump from your graph node to the code it generated, and the very useful ability to show a preview at any point in the shader code, via the generated code or within the nodes in the graph. And last but not least, I rewrote the actual graph system for HLSL/GLSL from the ground up so that it was more transparent and straight forward for engineers to understand and work with.

 

ShaderWorks was finally where it needed to be. I was always under the impression that since I did all of this development, studio engineers would be the ones integrating it into their own pipeline. I was wrong, developing the software from the ground up was just a small portion of my journey. I was later tasked with integrating ShaderWorks into Activision's biggest game at the time, Call of Duty Black Ops 1 and work on other tools and pipeline stuff. Integrating ShaderWorks into some foreign pipeline was one of the toughest things I did, but well worth the effort. It ultimately became our defacto shader authoring solution and was used day to day by over 20 artists and engineers, cranking out hundreds of really cool shaders for both Black Ops titles.

 

 

Shaders

 

I pretty much spent 4 years working on ShaderWorks, but during that time I was able to also do what I loved, and that's developing new and exciting shaders. Here's a few of the more involved shader tech I worked on in my spare time, nowhere near the amount of shaders I developed in my following 4 years, but pretty complex at the time.

• Developed what I think was the first GPU accelerated fur simulation using DX9 multi-pass render to texture point sprites with GPU skinning.
• Fully GPU-based tension mapping shader for dynamic wrinkles based on vertex/face tension. Later integrated it into the Call of Duty Black Ops engine and did the tension calculations on the CPUs/SPUs. The work was nominated for a BAFTA award.
• Developed a complex dynamic grass shader specifically for COD 3, was a huge hit at the studio and featured addition to their game.
• Pre-computed Radiance Tranfer ( PRT ) shaders for dynamic vertex-based image-based GI.
• All kinds of fancy HDR post-fx shaders
• Snow, rain, particle, water shaders.

Growing up I've always had a dream to run a 3D software company and after an exciting ride with Digital Immersion, I decided it was time to put my self to the ultimate test and give it a shot. I founded Mad Software and immediately began development of two software technologies ( Mad F/X and ShaderWorks ). While running Mad Software, I did everything from software, engine, shader, web-site and business development as well as sales, support, evangelism of my products and everything in between! Here is a bit of information on my software.

 

Mad F/X

 

Mad F/X is a high-speed real-time 3D animation / visualization application which was one of the first to use GPU acceleration and ASM shaders to give it a huge performance edge over other solutions. click for more info on Mad/FX...

 

 

ShaderWorks

 

ShaderWorks was to my knowledge, the first visual graph-based HLSL shader authoring application in the world. It was originally being developed as a tool to let artists create material shaders for Mad F/X without needing to know how to code. Surprisingly plenty of game developers and studios took interest in the tool and that's when I knew I had a killer app and decided to focus most of my attention on it's development.Visual graph-based shader development is viewed by many as the holy grail of shader authoring. Developing such a tool is a tough nut to crack and many companies and studios have tried again and again to solve this puzzle but have failed or came up short. ShaderWorks is now owned by Activision and was used extensively in two of the biggest game releases in history ( Call of Duty Black Ops 1 and 2 ) and a few others as well. When I left the company, I was developing version 6 and porting it over to DX11 for use on next-generation consoles like the XBox One and Playstation 4.

 

Running my own 3D software company was an amazing experience and I'm thrilled to have the oportunity to do it again through ShaderPlay LLC

While running my 3D software company ( Mad Software ), I did contract work remotely for Serious Magic ( a successful multimedia software company which has since been acquired by Adobe ). I was in charge of coming up with a library of shaders, as well as 3D rendering technologies for their upcoming products ( rendering engine, uber-shader system, and other custom special FX ). I also ported over their proprietary CPU-based vector / chroma keying algorithm ( used in their award winning Ultra products ) to a GPU-based HLSL solution resulting in a major performance increase. The company then moved me to California to work exclusively on their ground-breaking Ovation product which converts regular boring PowerPoint presentations into vibrant and interactive 3D presentations.

The owners of Solar Technologies acquired the rights to my real-time 3D software and rendering technology, flipped it into their new Digital Immersion company and had me continue development of a real-time 3D visualization application called MerlinVR which was ultimately bundled alongside ATI's All-in-Wonder graphics cards. I also worked on Merlin3D which was a 3D rendering application using the LightWorks rendering API. Aside from doing all the software development on MerlinVR, I spent a lot of the time on the road with the owners doing venture capital / investor meetings, showcasing the software and hitting up trade shows. Digital Immersion ultimately even received investment from ATI Technologies ( since acquired by Advanced Micro Devices ). I was young, full of energy and loved the excitement of playing a key role in a start up company.

I came into the company with a rare specialization in 3D hardware accelerated software development. I worked on their first windows-based Povray rendering software. The company quickly changed directions, acquired some of my intellectual property, rolled it into a new company named Digital Immersion Software and promoted me to Lead Software Engineer, and began development on a new product called MerlinVR and later Merlin3D.