Renowned Statistics Professor Bin Yu Calls Change in Career Direction "My Most Successful Failure"

After enduring 10 years of social upheaval, including the death of her father at the hands of the Red Guard, as a young girl living through Mao Tse Tung's Cultural Revolution, UC Berkeley Statistics Professor Bin Yu found both peace and order in a math textbook given to her by her cousin.

That immersion in math, coupled with an unplanned detour into probability and statistics, has led Yu to her current position as a Chancellor's Distinguished Professor in the Departments of Statistics and Electrical Engineering and Computer Sciences, and a member of the National Academies of Sciences. She is also the Class of 1936 Second Chair in the College of Letters and Science. Along the way, she has held positions at the University of Wisconsin, Yale University and Lucent Bell Labs.

She was three years old when Mao launched his effort to purge traditionalist and capitalist traits from Chinese society. Because her parents were intellectuals who worked for an agriculture university in Harbin, the Yu family was branded as "bad." Her parents were taken away by students who had joined the Red Guard and locked in makeshift cells at the university. She never saw her father again.

Not long after he died, Yu's mother was released and the family, including Yu's older sister and their devoted nanny, were sent, along with other university faculty, to a farm near the Russian border where they were to be re-educated by farmers. After spending their first night in a cramped room during a blizzard, Yu's mother and nanny pasted newspapers over holes in the ceiling to keep out the cold winds.

"During our time in Xianglan, I watched my mom in awe," Yu wrote in a book chapter about her life. "As the manager, she turned a troubled dining hall around in no time for the youngsters from the Shanghai region who were 're-educated' by the farmers. Her versatility was truly impressive, and her success had a lot to do with her warm and disarming personal style."

Yu began elementary school in a school set up in the other half of the dining hall, while her sister had to walk 45 minutes each way to attend school in the farm headquarters. But as the Cultural Revolution began to wane, the family was sent back to Harbin, where Yu's mother became a department chair at a university devoted to Chinese medicine and life was easier. It was around this time that she realized she was pretty good at math.

Mentors in math

"My cousin, Dawei Huang, gave me a book called Foundations of Applied Math," Yu remembered. "It was a refuge and a place where things were clearly right or wrong. I still remember the cover and tables."

Yu used the exponential and logarithm tables to convert numbers back and forth. "There was no computer, just the tables in the book," she said.

Her next mentor was Jianye Chen, who was hired to be a substitute math teacher in her middle school. Although she didn't have a class with him, he organized math competitions for the students, as well as a math club where they practiced polynomial factorization problems.

"My mom went to talk to him and asked him to tutor me after school, and he did," she wrote. "For the summer of my second year in middle school, he lent me a plane geometry book printed before the Cultural Revolution by Chunfang Xu. I did all the problems in the book that summer and loved it!"

Yu and her family moved to Beijing in 1978, but she returned to Harbin for the summer before high school and Chen taught her the entire high school math curriculum in one summer.

"That really put me ahead of my class in high school and set me on the path to become a math major in Peking University in 1980," she said. "I put PKU math as my first choice on my college application in order to realize Teacher Chen’s dream – he was not able to go to PKU because his father was labeled a 'rightist' in the 1950s."

Looking east to America

While at the university, she was visited by her uncle, who had been in the U.S. before World War II and was persuaded to enlist in the U.S. Army (or face deportation) and served as an interpreter in China during the war. Afterward, he attended Harvard on the GI Bill and had a long career with IBM.

On his visit, "He gave me my first bicycle. At the time you needed a special permit to buy a bicycle," Yu said. At a store where items could only be bought with foreign currency, he came out with a "beautiful blue bicycle -- almost all the bikes in China were black." The bicycle provided her transportation during her years at the university.

Chen, her former teacher in Harbin, urged her to attend graduate school in the United States. With that in mind, Yu continued to study English on her own and with friends after completing the required two years of English instruction.

Yu placed first in the math parts of the entrance exams for graduate school in math at Peking University, but the professor she hoped to work with would not accept her. Because of that, she decided to switch from math to probability and statistics. Although she considers it a forced move, it has allowed her to do applied work in areas such as neuroscience, genomics and precision medicine. "I call it my most successful failure," she said. "I feel I was rejected because I was a woman and he didn't think I would go very far.”

Heading to Berkeley

Her opportunity to study in the U.S. came in the form of the Chern Math Exchange Program between China and the United States. The program was named for Shiing-Shen Chern, a mathematician born in China who also taught at UC Berkeley and was the first director of the Mathematical Sciences Research Institute in Berkeley.

In the beginning, she was not nominated to participate in the Chern program by Peking University, although four male students were. But she got her chance when some students from other universities did not pass the on-site interview at her university. She was interviewed in China by two members of the American Mathematical Society (AMS), one of whom was T. S. Yau, a Harvard professor. Yu  was one of 15 students accepted in 1984 and with a recommendation letter from the AMS she applied for and received a fellowship from UC Berkeley. Prof. Lucien Le Cam was the admissions chair that year and he and Yu corresponded often and Le Cam sent Yu her first statistics book by Bickel and Doksum.

"He was my mentor from day one," Yu said. She continued to study English throughout the year before heading to Berkeley in August 1985. Despite her studies, she found the first year of speaking English a rough one and life in the Bay Area was full of new discoveries.

"I had $500 in my pocket and one friend from the math department at UCB met me at SFO," Yu remembered. "On the drive up 101 I thought the sky was so much lower than in China, because of the clouds."

She first stayed with this friend from the math department in a place on Regent Street as she looked for her own apartment. She learned how to open a checking account and find furnishings at garage sales for her new home on MLK Way that she shared with a friend in biology who worked nights in a lab at UCB. She borrowed her friends’ car to learn to drive and on the day she barely passed her driver's license test, she parked the car near campus and left the car lights on. When her roommate went out to the car to come home that night, the battery was dead. Lesson learned.

Yu continued her studies, making friends with other students from around the world. A new faculty member, Terry Speed, arrived in 1987 from Australia and became her co-advisor with Le Cam. By that time, she had already extended some results in empirical process theory from the independent case to the dependent case under the supervision of Le Cam. Her research with Speed took a different direction into the minimum description length (MDL) principle at the interface of information theory and statistics. Moreover, she credits Speed with extensively mentoring her on critical thinking skills and applied statistics. Her interests in science grew as well, especially through her many conversations with Speed.

She also played soccer with the statistics team called “Bootstrappers,” and found she was better suited to defense. In 1987, she returned home to marry her boyfriend and a year later he joined her in Berkeley. Thinking they would return to China, she didn't worry about finding a job in the U.S. But when she finished her Ph.D. in 1990, they made a last-minute decision not to go back.

Instead, she got a position at the University of Wisconsin in Madison. After a year, Berkeley called and she interviewed and accepted a position. But she stayed on at Madison for another year so she could fulfill her commitment to spend a semester at Yale as a visiting professor before returning to Berkeley.

In 1996, Yu was up for tenure but it didn't go smoothly. The Friday before the Monday meeting on her tenure case, a letter was put in all voting faculty’s mailboxes with four signatures urging them to deny Yu tenure. The vote was close -- 11 to 9 in favor, with two more votes in favor coming later. She was pleasantly surprised when campus approved her tenure very promptly in February 1997.

Feeling somewhat bruised by the experience, Yu took leave to work at Lucent Bell Labs in New Jersey.

"I found it very liberating and we could pursue our own ideas," Yu said. "That's where I got into machine learning, and started collaborations with Mark Hansen on MDL and Peter Bühlmann on machine learning."

Having kept in touch with her cousin who gave her the math book about 25 years ago, Yu also found an opportunity to  co-author a paper with him. Now back on campus for 20 years, Yu and her group have spent the past seven months on a COVID-19 project that builds on predictive work that she did with her cousin and two others at Bell Labs.

She has also authored a paper and is now finishing a book describing a framework that she believes will improve the field of data science and lead to a more rigorous and trustworthy use of methods such as machine learning. Yu laid out her framework for integrating predictability, computability and stability, which she calls PCS, in her paper  "Veridical data science" co-authored with her former student Karl Kumbier (now a postdoc at UCSF) and published in the Proceedings of the National Academy of Sciences in February 2020. Read more about the PCS framework.

She and Rebecca Barter, Yu’s former student and current postdoc, are now adapting the material into a textbook to be published by the MIT Press in 2021. They also plan to make the material available online at no cost.

Helping Shape Data Science at Berkeley

In addition to her work to develop and propagate her work to develop the PCS framework to make data science more trustworthy and transparent, Bin Yu has also played an important role in establishing UC Berkeley’s data science program.

In 2014, she was a member of the campus’ first data science education committee. The following year, she chaired the Statistic Department’s internal review committee under department chair Michael Jordan as part of the department’s 10-year external review. That process produced the first written document proposing a stand-alone school for data science.

To bring data science to the classroom, Yu and three of her colleagues Joseph Gonzalez and Joseph Hellerstein (both Computer Science) and Deborah Nolan (Statistics) co-created and co-taught the first instance of Data 100 in Spring 2017.

Yu served on the faculty advisory board chaired by Cathryn Carson that wrote a report outlining the intellectual and organizational vision for today’s Division of Computing, Data Science, and Society (CDSS). She helped bring CDSS to fruition as a member of the faculty advisor committee for Interim Dean David Culler. Yu was also a member of the hiring committee that selected Jennifer Chayes as the Associate Provost for CDSS and Dean of the School of Information.

~ Jon Bashor