Alexa
Product Design | How we set a new course for the design of a companion app for Amazon's Echo product line.
In early 2014, I came onto the Device + Services Design Group (DDG) team within Amazon, working as a UX Production Designer to create Visual Design Specifications for Engineer teams to build out Fire Tablet UX for the family of hardware.
In mid-2015, working with my manager, we came up with the idea of quality control for UX—we called it “Visual QA.” Before creating the Visual QA team, Designers would review builds for their respective programs. Still, it was only after the Designers’ features or domains were considered “finished” and would have a window of time to review a build and file bugs against anything that wasn’t aligned to UX POR. This would result in only a small number of the issues uncovered being fixed before the build was released to our customers.
My focus on Insisting on the Highest Standards and dedication to Customer Obsession, two of Amazon’s Leadership Principles, were ideal for heading up this new team division. The Echo Show and Fire Tablets were the first products to benefit from my services.
In the first 8 months, capturing 74 issues to help align the software to the intended UX design. Collaboration with Engineering to resolve 71 of the original problems found brought great success to this review process and expanded efforts toward FireTV.
+78%
Bugs filed and fixed since 2015
15×
Device coverage since 2015
+50
Domains reviewed across 20+ devices
+100%
Focused on end-to-end experience for customers
FireTV and the Alexa App were added to the Visual QA workflow in the latter half of 2016 and into 2017. We found a need to bring on an additional head to allow the review efforts to remain strong and not overlook the devices that benefit from such a review. I captured 976 new issues seen across various devices during this time, including the Echo Spot, FireTV Cube, FireTV Edition, the Alexa App, and Fire Tablets. I worked closely with UX and followed up with Engineering to ensure the most egregious issues were addressed before launch dates, resulting in 830 of those original bugs being resolved.
In 2018, we grew the Visual QA team. We were fortunate to find an individual who was just as passionate about UX POR. By doing so, the two of us were able to take on a massive review across the entire ecosystem of hardware to align the UX and make the Amazon experience cohesive with Echo Show, Tablet, and FireTV. This resulted in 1810 issues being captured across the Alexa screen-based ecosystem. That was a 600% increase in effort compared to 2015 with the first review. While we worked with UX and Engineering to bug bash and prioritize the Blocker and Critical issues before launch, 1662 issues were resolved by the end of the year. This showed that the Visual QA team needed to scale to support the growing number of first-party devices.
With the continued effort through 2019, we were able to start reviewing Alexa Design System (ADS) components and templates. This requires a higher level of activity due to the complexity of needing to review the content on the Web Host (Authoring Tool) and the Viewport Profiles. This dual effort is crucial to ensure that what the developers build within the Web Host will match what is pushed to the devices. In that year, 805 bugs were filed against this program alone.
We reference Visual Design Specifications (POR) created by our Production Design team and compare those specs with on-device screenshots from referenced builds. We report any visual defect as a bug to Engineering via JIRA. After bugs are filed, we work with Engineering to verify the implemented fixes.
As seen in the images below, we would overlay screenshots from the nightly builds with a modified UI file commonly referred to as a “red overlay.“ We can easily show missed alignment and font styles by using this method.
In the traditional sense, Quality Assurance is typically focused on the functionality of the software—a button clicked makes the expected action. With our Visual QA team focused on the aesthetics and UX patterns, we did familiarize ourselves with Android code and internal tools developed by our Design Technologists. We can more easily update devices with the latest builds and control interaction models to more easily inspect areas that might otherwise be missed due to the short time on screen. During a review, software we continued to utilize:
While we have found ways to use UX software like Sketch and Figma to gain a form of automation in Visual QA, we have utilized ticket template creation through Jira to create a one-click approach to cutting tickets against regularly reviewed domains.
In 2020, we worked to build out sites within our team’s page to help record reviews, bugs filed, and the program health and status based on the known release dates. By doing so, the stakeholders for each program can quickly reference these sites to know if any issues remain to be addressed that could risk launch for a particular program.
One of the Visual QA team’s privileges is access to the working files of the Visual Specification documents created by the UX Production Designers. With this privilege, we can quickly pull the layouts we need to review and crosscheck the screens from the build.
In addition, the UX Production Designers keep Design Systems up-to-date that are used within Sketch and Figma. These are built for the Design teams’ benefit, but the Design Systems allow Visual QA to focus on reviews centered around the components and templates (e.g. Alexa App).
Over the years, the requests to use our services against other Amazon devices have allowed our team to easily add them into our testing practices. Still, in other cases, we have explored new ways to offer our services as our team of two would find we would have to tag-team reviews. We have been successful through our teamwork and supportive nature.
In some cases, we have known that the requests can be within a Sprint that we’re already in. These reviews are usually only a few screens, but they still are somewhat tricky for us to provide our valued attention. So we offer a “high level” review approach that explores the layouts against the Visual Specs to address and call out any critical or blocker issues that are visually a risk to our Customers seeing it on their devices.
Since 2015, Visual QA has grown to review over 40 devices (including multi-generational devices), covering over 50 domains across the Amazon family of products, and working on reviewing 3 major design system upgrades for the Echo Show family of hardware.
In addition to this, we continue to learn and develop best practices around what Visual QA means to Amazon.
As of late 2021, the Visual QA team has been exploring ways to automate efforts as the design systems, and device library continues to grow. Not to mention looking at ways to include reviews for accessibility standards and string validation.
Do amazing products excite you just as much as me? Reach out