Can You Do UX Research On a Thing You Can't See? Can You Do It In a Week?

The first most frustrating thing that can happen to a User Experience professional is for all aspects of user research and usability to be summarily cut from a project budget.

The second most frustrating thing that can happen to a User Experience professional is for there to be budget for research and usability, but attached to a absurdly short timeline.

The third most frustrating thing that can happen to a User Experience professional is to not have access to the site for which the research and usability is to be conducted.

In the 4th quarter of last year, I worked on two projects that met the second and third criteria, simultaneously. This was my first time working with this agency, and both my project teams were remote. We were all voices on a phone to each other. 

I did not go crazy. I did not yell. I got the work done, and both the clients were pleased.

Let me tell you the story.


Wait: why aren't there any images of deliverables in this case study?

A couple of reasons:

  1. The work for Client #1 was for an intranet. Intranets are, by definition, proprietary. I wasn't even able to access the site, and had to work from a handful of screen captures, and PowerPoint mockups. (Typically, access to internal client sites require separate, extensive, paperwork and additional background screening that can take at least 2 weeks if not longer. Because I was a contractor, and I'm sure that the agency had no interest in letting the client know that, and the deadline was insanely tight, this was never an option.) 

  2. Client #2 was a company that the agency I was contracting with was incredibly eager to develop a long term relationship with. As such, it was a clear cut case of, "Just get it done, and make them happy." The goals and purpose of the usability work were never shared with my team. 

And this is reason #3.

And this is reason #3.

    EOY Budget That Must Be Spent

    Both projects were driven by the typical enterprise requirements that whatever budget is not spent, is lost—and in most companies that also means the department takes a negative hit to next year's budget as well. 

    Client #1 was a huge, international personal care corporation. They were about to move their intranet to a brand new platform, and needed help crossing the finish line. The team was internal, tiny, and trying to complete this work on top of their ordinary workloads. The project had to be done by the end of the year.

    Client #2 was a huge, multinational conglomerate. They had recently overhauled their internal career path training and documentation, a combination of clickable PDFs, and content from a third-party provider. The team was internal, tiny, and seemed to be driven by powerful, yet mysterious, political factors. They wanted usability testing. We had one week to do it—in December, before the end of the year.

    Client #1: Balancing My Employer With the Client

    I was a UX Lead contractor at a digital agency for both projects. What this has always meant in practice is having two clients: the agency, and the agency's client. My job, always, is to meet the agency's client's needs, of course, but also to balance the requirements the agency has of me. 

    And sometimes, these things are in conflict.

    Client #1 was very clear in all the documentation provided that they did not want research. They had conducted all the research they needed. Based on that research, they had provided the agency with a PowerPoint outlining, in significant detail, what pages need to be designed, the navigation, and what content to put on each of the pages.

    In short, we were there to color in the lines. I've worked on these kinds of projects before, and while they aren't the most rewarding, you realize that you're there to take the load off your client, and help them look like heroes. Our designs, and the move to the new intranet platform, were also supposed to be so fantastic that it would help the client team convince other departments to also moved to the new platform.

    Again: we're there to help them look like heroes. Easy peasey.

    During the internal team kick off call, the agency team expressed a desire to 'do the right thing', and press for a longer project, with more research, and time. After all, they reasoned, intranets are complicated, and we should make sure that it gets done the right way. 

    They were also very keen to establish an ongoing relationship with this client.

    The challenge I was facing was that the client had been very clear: no research. There was neither time, nor budget, for the 3+ weeks of user interviews that the agency team had expressed interest in (e.g., 'the right way') during the internal kick-off. However, the PM asked me to lead the discussion with the client, and see if we could bring them around to our way of thinking.

    I had a couple of priorities: 1) to find out if we could even get access to the intranet in order to dig around and understand it (spoiler alert: we couldn't); 2) figure out how to gently, but firmly, help my employer, the agency, hear the client's desire to move forward without research; 3) assure the client that we understood their 'no research' request, but we just had a few clarifying questions to ensure that we had all the information we needed to move into wireframing.

    In short: thread a needle in a room lit only by candlelight.

    I am an ace improvisationalist when it comes to working with clients. I make small moves, wait for reactions, and then, more small moves. Though I had some fairly solid assumptions entering into the client meeting (spoiler alert: the assumptions were correct), the small moves allow me to adjust on the fly as needed. I listen, I process, I adjust. If something catches me off guard, I'm more apt than not to note it, a la, "Wow, that's a little surprising, can you tell me more about XYZ?" 

    Through a series of gentle, small questions, as well as re-stating what the client said, I was able to make it clear to the agency that this project really wasn't a huge intranet redesign undertaking, and the client was able to clarify their needs. 

    You Just Get Comfortable With Being Wrong

    As for how you do wireframes when you can't see an intranet: you just do the best you can. I had never worked with this particular platform before, though the agency team was well versed in it. I asked a lot of questions, and I was okay with being wrong. I had a list of what they wanted, so I worked with that. 

    By being wrong, and getting feedback, and talking, you eventually arrive at the right place. The wireframes were approved, and we moved into visual design. And guess what? We were still wrong. The client asked for another design exploration, which wasn't in scope, but we worked it through. 

    So, the intranet site has gone live with the new homepage and landing pages we designed. I will never see it. That's just how things are. 

    Client #2: The World's Fastest International Usability Test For A Site I Can't Access

    There was a lot of confusion. The agency team had the site log in at one point, but it stopped working. The client didn't know, or didn't understand, that we didn't have access to the site. In any event, our team (myself, a content strategist, and a PM) were pretty unnerved: one week to conduct international web conference usability tests, half on clickable PDFs, half on an IP-restricted intranet training site. We had a sample batch of PDFs, which we didn't really understand the purpose of, and no clue what the web site was about.

    We agreed that we would raise, gently, our concerns with the client, and try to push back on what seemed like a totally random date.

    Spoiler alert: it didn't work. Moreover, the invitations with the usability participants were already flooding in, for the following week. 5, 12 hour days in front of me and the content strategist.

    We spent about 20 minutes of the call on WebEx as one of the client team walked through the training web site. But it's just not the same as having the time to explore it on your own. There also wasn't enough time to ask in depth questions about the PDFs. We at least managed to get the client to agree to have one of their team members on every call so that they could navigate to the website on the WebEx, allowing me to see the site and track the progress of the participant through the tasks. 

    This was the single most bizarre experience of my career. All right, I thought, fine. Let's just wing it!

    I opened up OneNote and typed out a quick and dirty usability script—all very high level questions that would at least allow me to improvise—and shared that with the CS and PM for comment. They were just as much in the dark as I was, so they sent back thumbs up. The CS wanted to dig a bit into brand perception, so she added a section of questions for that topic. The CS and I were going to be running the sessions, about a 90% me/10% CS split in terms of responsibility

    The fear was simple: we didn't know WHY we were conducting these tests, so we had no idea what we were supposed to be looking for. To our eyes, the PDFs were not ideal: long, verbose, and awkwardly linked, their purpose as a tool for career planning was opaque. We knew the site had some sort of multimedia, but we wouldn't know more until we were 'walking' a participant through the site (read: when we noticed a link and decided to ask the person to click it). 

    The participant pool was weighted heavily toward Europe, the Middle East, and Asia. Overall the interviews went very well, and we were able to continuously improvise, and adjust questions as we went. We asked questions in order to understand the context of these materials: how did people plan their careers at the company? We learned a lot of things that weren't surprising (a lot of DIY, personal networking) given the labyrinthine nature of the documents in front of us (3, 50+page PDFs for each department, explaining the variety of competencies, and career paths, available to employees). 

    We only ran into one serious roadblock: an employee in China had very low English language skills, and we learned that she thought this was going to be a training session for her. Then, after I explained what we were doing, she thought we were testing her. At this point, I felt terrible: she was confused, and we weren't going to bridge the language divide, so I thanked her for her time, and we ended the call. Overall, we learned that, world wide, employees were routing around the overly complicated career planning process in remarkably similar ways. 

    The week went by in an absolute blur. I put together a deck with the findings, and shared that with the CS and the PM by Wednesday of the following week. Since all of us remained in the dark about the real goals of the project, we didn't know how the deck would be received. 

    To our extreme surprise, and relief, the client was really receptive to the findings when we presented on Thursday. We advocated for moving the PDFs onto the web, to take better advantage of the interconnected content. The training course site had huge issues with repetitive and vague titles for content, as well as a lot of clicks to get to said content. Oddly, the lead client was not on the call, and the budget literally ran out the day of our presentation, so our report went out into the void. 

    Such Is the Freelancing Life

    These are some of the more extreme examples of projects I've been a part of throughout my time as a freelancer. Most projects are pretty typical: you walk in, the SOW is signed, the scope is fixed, timeline set. You come in and do what the line items say need to be done. 

    What I've learned is to be flexible, and pay attention: you listen to your employer, your team mates, and the client. You look for opportunities, as well as for potential issues. You have to be able to see the future a little, and that's something that only comes through experience. Once you've lived through a couple of troublesome projects, you've got a toolkit to use to help yourself and your team get through the next one.