The Global Experimentation on NgAgo

Kate Qin Zhao

9 min readSep 30, 2016

The Global Experimentation on NgAgo

By Kate Qin Zhao, PhD

Drafted on 2016–09–26

Social media has changed the way we conduct scientific research, especially in how we share information and results.

The “social media factor” has been magnified this summer of 2016, by the global replication effort of NgAgo.

For many scientists around the world, the summer of 2016 has been a roller coaster ride.

They were motivated by the potential of NgAgo for genome editing, quickly replicating it in their own labs, but only found disappointment when no one could replicate the results as originally described in the Nature Biotechnology (NBT) paper published in early May 2016 (1).

Because there are so many replication failures, the “hot topic” of NgAgo has cooled down quite rapidly, as reflected in Google Trends.

Scientists have been actively posting their thoughts and results for NgAgo on various social media platforms ever since May’2016, when the NBT paper was available online. Most of these posts are fragmented pieces scattered here and there, quite confusing for the public to fully grasp the complete picture of this event.

To have a clearer picture of this global NgAgo replication effort, I have collected as much as possible publicly available and relevant information from various social media platforms, and summarized them to help “visualize” the scope of this historical “social media” science event.

Before going into the details, let’s first review the backdrops of this event.

• What is so special about NgAgo?

• Why has it attracted so many scientists around the world to replicate it in their own labs?

It all lies in the potential for genome editing, the possibility to edit any living organism with ease. This is a field that has experienced so much advancement in recent years, especially after the introduction of the CRISPR/Cas9 technology (2).

Genome editing with CRISPR/Cas9 today is almost like a parallel to PCR with Taq DNA polymerase invented some 30 years ago. Both of these two technologies are truly disruptive. They have led to the explosion of applications ever since their first introduction to the science community. The way we conduct biological research, and to some extent, our “life style” in general, have forever been changed by these technologies.

For PCR, after Taq DNA polymerase, enzymes with higher fidelity, such as Pfu and the like, have been introduced. These new enzymes have continued to fuel the growth of basic research as well as the application of PCR, such as in various types of disease diagnostics and forensic tests.

For Genome Editing, the CRISPR/Cas9 system is so robust that it has enabled so many labs to enter the field, and its potential has also attracted many high caliber investors.

NgAgo/gDNA system is trying to ride this genome editing wave, and it has been introduced as a higher fidelity contender of CRISPR/Cas9 with similar efficiency and less constraints. These features have naturally induced great interest from researchers around the world.

Will NgAgo hold on for Genome Editing just like high fidelity DNA polymerases for PCR?

Apparently not, as judged by the failure in its replication attempts, all of which happened in a short stretch of four months.

In fact, frustrations with NgAgo have surfaced as early as just one month after the NBT paper was first available online. Researchers have shared their “negative” results on the internet, such as in google groups, on twitter, in discussion forums (eg, mitbbs/biology), and on many other online social spaces.

With social media, words can really travel fast.

More frustrations have been reported as an Addgene blog posted results from a Google Survey (3).

As of August 1st 2016, out of 165 survey responses, 88 tested NgAgo, only 1 said “Yes” to successful replication of NgAgo for genome editing.
As of September 19th, 105 out of 211 responses failed to replicate NgAgo in genome editing, while 10 said “Yes”.
In a curious incidence between the two data collection dates, someone voted “yes” multiple times all within a 15 minutes time frame.
Adding to the complexity, most of the “yes” votes, including the initial one, were claimed to be just “a test” on the survey itself rather than from real replication efforts.

So, unfortunately, these “positive” responses from the survey have to be disqualified and classified as “false positives”.

Nature magazine followed suit by publishing an editorial on the replication issue, about three months after the original NBT paper was published (4). In this editorial, three anonymous Chinese scientists claimed to observe positive results of NgAgo for genome editing, but unfortunately again, no supporting experimental details have been released by them since the publication of the Nature editorial; therefore, these three cases have to be classified as “false positives” as well.

Other anonymous claims of “positive” NgAgo replications have been posted on various discussion forums; but yet again, no supporting experimental details could be found for any of these claims. In addition, most of the sources of these anonymous “positive” cases somehow have always been mysterious and hard to track down. Therefore, these claims have also been disqualified in this summary.

Amidst the recurring reports of replication failures, on August 8th 2016, the corresponding author of the original NgAgo genome editing paper, Dr. Chunyu Han, released an updated protocol on Addgene (5), implying that the original protocol published in their NBT paper might not work well for others.

One has to wonder, why did they publish the original protocol to begin with if there were known issues associated with it? Also, how did they manage to achieve the results published in their NBT paper?

Unfortunately yet again, the modifications listed in this updated protocol only raised more questions rather than provided answers to the replication failures.

Not unexpectedly, replication attempts with the new protocol failed just as much as with the original protocol.

To make the situation even more confusing, to date, Dr. Han’s team have not been able to provide any successful replication results by themselves either.

From the available information on NgAgo collected from online sources, a trend has emerged:

For all “successful replications”

The source of the information could not be identified and/or have been found to be invalid
Key experimental details are missing

For all validated “failed to replicate” cases

The source of the information can be identified and/or found to be valid
Key experimental details available

“Failed to replicate” is the characteristic of all the valid cases collected so far for NgAgo.

To understand the scope of this global “failed to replicate” effort, let’s group the information into four categories:

1. Interest of NgAgo by Region

– Summarized from validated cases

The NgAgo replication experiments have been conducted by scientists from around the world.

US, Europe, Australia, China, Japan, India

Note: Addgene NgAgo vector has been reported to be distributed to nearly 30 countries (6)

2. Detection Methods for Genome Editing

Scientists have really worked hard for NgAgo, many well-established detection methods for genome editing have been utilized.

T7E1
Surveyor
Sanger sequencing
NGS
Phenotype detection
And other validated assays for genome editing

3. Systems tested

NgAgo has been tested in many model systems without success:

Mammalian cell lines (HEK293, Hela, etc)
Mouse/human NS cells
Yeast
Fruit fly
C. elegans
Zebra fish
Mouse zygote
And, maybe even more…..

4. The cost of this global experimentation on NgAgo

Estimated to be at least $2.25–$2.5M USD

This is a rough estimate based on the following information:

a. Number of labs performed NgAgo replication experiment

~400 vectors distributed by Addgene up to Aug 2016 (3)
Also, even before its availability from Addgene, quite a few labs acquired NgAgo vector through other channels, such as directly from Dr Han’s group, by gene synthesis, etc. A conservative estimation is about 50–100 labs
So, there are at least 450–500 labs interested in testing NgAgo since May 2016.
Assuming 50% of these labs completed the replication work (based on statistics from the Google Survey(3), ~225–250 labs have completed NgAgo replication experiments.

b. Expense per lab

Each lab’s investment in NgAgo is estimated to be about $10K USD

(materials/reagents, personnel time/effort, etc)

Therefore, conservatively, scientists around the world have invested about ~$2.25–2.5M USD in NgAgo, all invested in a short period of four months (mid May to mid Sept’2016).

To conclude from the information collected so far:

This is a global experimentation on NgAgo for genome editing; scientists from many countries have contributed budget, time and effort for it.
To date, there is no valid, publicly available report of “positive replication” of NgAgo for genome editing
Social media has changed the way we conduct research.

o Words really spread fast!

o Many scientists have delayed their NgAgo replication efforts after failures and frustrations surfaced on the internet.

Here are the questions that remain to be answered:

What has really happened with NgAgo for genome editing?
Why have Dr. Han’s team not come forward and provided any positive replication results themselves?
Who is responsible for such cases of the wasted time, effort and budget
How to avoid this type of situation in the future?
How can we maintain scientific integrity amidst everyday information overload?

Finally, what will eventually happen to NgAgo as a genome editing tool?

For scientists who have participated in the replication effort

They have already wasted time, effort and budget.
NgAgo seems to have already sunk to the bottom of their “useful tools” list.
They are lucky to still have CRISPR/Cas9, a robust, reproducible and efficient genome editing technology that will be here to stay.

2. For other scientists and the scientific community

They probably will ignore NgAgo for now and keep on using CRISPR/Cas9
The general interest on NgAgo for genome editing has already dwindled down significantly

o Initially, the replication issue of NgAgo has led to quite a few reviews, such as articles on GenomeWeb (6) and Biotechnique (7), all published before the end of August 2016

o Since September 2016, the topic seems to have cooled down significantly, with much less information posted on social media outlets, also, with much less media coverage.

3. For Dr. Han’s team

They hold a patent on NgAgo for genome editing (8)
There are reports that they have been awarded over 200 million RMB (equivalent to over 30 million USD) from the Chinese government to set up a genome editing center, therefore, they seem to have plenty of financial support for future efforts in the improvement of NgAgo for genome editing as well as in its applications (9).
However, no matter what types of research and technology development may be generated by Dr. Han’s team in the future, one thing for certain is that, if Dr. Han’s team does not come forward and give the scientific community a satisfactory explanation on why their work could not be replicated, a big question mark will always be hanging over any of their scientific endeavors.

Through this global replication effort of NgAgo, we can see more clearly that, the way we share information on scientific research has forever changed in the age of social media.

Social media has become a great vehicle to quickly promote a scientific discovery/invention. But, it also could be a double-edged sword. Once the message is out, if there is a problem, hiding it from the public will no longer be easy.

Acknowledgement

Thank you for all who have kindly shared their experience and experimental details with the author.

References (Links)

1. NgAgo NBT paper — The article in question

http://www.nature.com/nbt/journal/v34/n7/full/nbt.3547.html

2. CRISPR/Cas9 technology

https://www.ncbi.nlm.nih.gov/pubmed/22745249

http://science.sciencemag.org/content/339/6121/819

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3712628/

3. NgAgo Google Survey Summary

http://blog.addgene.org/google-forums-round-up-first-impressions-of-ngago

4. Nature editorial