Skip to content Skip to navigation

Overview

Recent news:

1. Currently recruiting a Lab Manager to help run day-to-day operations in Prof. Shenoy's and Prof. Henderson's NPTL research group and in Prof. Shenoy's NPSL research group (appointed via HHMI at Stanford). url

2. Recent presentation to a general audience, covering some of our recent research: Shenoy KV (11/5/2021) Brain-to-text communication via imagined handwriting. 2021 Tencent WE Summit, Beijing, China. 32 minutes. YouTube (GreenScreen)

  • Wall Street Journal article url
  • Financial Times article url
  • Global Times article url

3. Recent overview article on BCIs / BMIs for a general audience, spanning basic academic science and engineering to emerging industry and commercialization efforts: Regaldo A (2021) A computer mouse inside your head. MIT Technology Review. Nov/Dec 2021 issue: 28-35. pdf url


We conduct neuroscience, neuroengineering and translational research to better understand how the brain controls movement, and to design medical systems to assist people with paralysis. These medical systems are referred to as brain-computer interfaces (BCIs), brain-machine interfaces (BMIs) and intra-cortical neural prostheses.

We conduct this research as part of our Neural Prosthetic Systems Lab (NPSL), which focuses on fundamental systems and computational neuroscience and neuro and electrical engineering, and is directed by Prof. Krishna Shenoy.

We also conduct this research as part of our Neural Prosthetics Translational Lab (NPTL), which focuses on fundamental translational research with people with paralysis as part of clinical trials, and is co-directed by Prof. Shenoy and Prof. Jaimie Henderson (Dept. of Neurosurgery).

Neuroscience.  Our neuroscience research investigates the neural basis of movement preparation and generation using a combination of electrophysiological, optophysiological, behavioral, computational and theoretical techniques (e.g., dynamical systems, recurrent neural networks, dimensionality reduction, single-trial neural analyses). For example, how do neurons in premotor and primary motor cortex plan and guide reaching arm movements?

Neuroengineering.  Our neuroengineering research investigates the design of high-performance and highly-robust BCIs. These systems decode neural activity from the brain into control signals for restoring lost motor and communication abilities (Shenoy & Yu (2021) Chapter 39, Principles of Neural Science, 6th ed.). This work includes statistical signal processing, machine learning and real-time system design. For example, how can we design BCIs rivaling the communication rate of spoken language.

Translational.  Our translational research builds from our clinical trial (BrainGate2: Feasibility study of an intracortical neural interface system for persons with tetraplegia (NCT00912041); BrainGate.org; Timeline of selected BrainGate publications, url) and is conducted as part of the NPTL. For example, how can we bring new neuroscience, neuroscience, measurement technology and machine learning together to help people with paralysis in real-world settings?


Fig. 1. Motor cortical areas, array recordings and CTD-DSF analyses and visualizations. (a) NHP PMd & M1d and (b) human PMd & M1d cortical areas (but see Willett et al. (2020) Cell where gyral M1d appears considerably more like NHP PMd than M1d) involved with arm, hand, finger, face and speech movements. (c) Motor homunculus (Penfield, 1950). (d) Participant T5’s MRI with PMd (magenta ■) & M1d (cyan ■) arrays, and PMv (yellow ■), M1v (blue ■) and IFG-Broca’s area arrays (green ■, red ■) for speech. (e) Center-out task with cursor (white) controlled by a haptic-robot position. (f) On-screen cursor trajectories. (g) Utah (NHPs, people) and Neuropixel (NHPs) electrode arrays (first Neuropixel recordings in NHPs, Trautmann et al. (2019) Neuron). Many Neuropixel electrodes measure each neuron (g, right), even during electrical microstimulation (μ-stim, purple pulses; ERAASR, O'Shea et al. (2018) JNE). (h) Neuropixel current source density (CSD, Fig. 1g-left), white matter 'positive-firstpeaks' and "Kilosort" analyses ascribe neuron identity, putative cell type and approximate cortical layer (Fig. 1g-middle). The highest firing rate direction (preferred direction, PD) and modulation depth reveal random PDs (Fig. 1g-right), which differs from the prevailing view. (i) Dynamics and observation equations (Shenoy & Kao (2021) Nat Comm). (j) Neural population trajectory (Churchland*, Yu* et al. (2010) Nat Neurosci), which typically resides in a lower-dimensional (D) space (10-20D; 2D shown) than the number of neurons (∼200; 3D shown). (k) Trajectories evolve according to the net effect of A, B and u (gray vector flow field). (l) shows single-trial hand trajectories to 7 reach targets. (m) shows reach-direction averaged trajectories inferred for each of 44 recording days (with LFADS, Pandarinath et al. (2018) Nat Meth) projected into the subspace spanned by the condition independent signal (CIS, global rise in firing rates in PMd/M1d; Kaufman et al. (2016) eNeuro) and the first jPCA plane (jPC1, jPC2, where jPCA extracts rotatory dynamics; Churchland*, Cunningham* et al. (2012) Nature, Sussillo et al. (2015) Nat Neurosci). (n) shows single-trial neural trajectories from within a session day.

The human brain comprises approximately 100 billion neurons, each with approximately 1,000 connections to other neurons. How do these neurons coordinate to control movement? To discover fundamental principles we conduct electrophysiological experiments with nonhuman primates (NHPs) and people, and create theories and computational analyses to decipher how computations are performed through their collective dynamics (Computation Through Dynamics, CTD; Vyas et al. (2020) Ann Rev Neurosci). Starting in 2006 we proposed a 'Dynamical Systems Framework' (DSF; Shenoy et al. (2013) Ann Rev NeurosciShenoy & Kao (2021) Nat Comm[1], inspired by advances in engineering and by Laurent, Fetz and colleagues, to place studies of cortical motor control on a more mathematical footing. Success depends on the ability to relate single-trial behaviors to single-trial neural activity, estimated on a millisecond timescale. The instantaneous status of neural activity (action potential firing rate) across neurons is termed the 'neural population state,' and the ms-by-ms evolution of this state, which is governed by dynamics arising from synapses and circuits, is termed the single-trial 'neural population trajectory'. The CTD-DSF has shifted how hypotheses are generated and tested, and many groups studying various neural systems have adopted and advanced the CTD-DSF. It has revealed several mechanistic 'dynamical motifs' which appear to be conserved across many areas, tasks and species (mice, NHPs, humans). Our overarching hypothesis is that dynamical motifs are an essential computational building block. The CTD-DSF has helped move systems motor neuroscience from somewhat qualitative and representational interpretations of individual neurons to a quantitative and computationally-mechanistic formulation of neural populations, and does so in a way that bridges to cellular and molecular mechanisms.

We have pursued a deeper understanding of the cortical control of arm movements, motor learning, decision-making and BCI control [2]. We also paired the CTD-DSF with human electrophysiology focused on understanding rapid, dexterous, multi-limb/finger movements and speech, which is only possible with people. We have uncovered new encoding principles (Willett*, Deo* et al (2020) Cell) and these discoveries enable new neuroscience studies and classes of BCIs: a "Brain-to-Text" attempted-handwriting BCI (Willett et al. (2021) Nature) and a "Brain-to-Speech" attempted-speaking BCI (Stavisky et al. (2019) eLife). This 'only possible in humans' neuroscience helps highlight and prioritize new opportunities for next-generation basic science, in NHPs and in people. This human research is a collaboration with Henderson (at Stanford) & Hochberg (at MGH/Brown and across sites) who direct the BrainGate2 pilot clinical trial (NCT00912041).

NHP and human cortical areas involved in motor control. Despite the importance of understanding the principles and mechanisms underlying motor control, many fundamental questions remain elusive.  Due to their centrality in dexterous-movement control, we have focused on dorsal premotor (PMd) and primary motor (M1d) cortex in the arm, hand and finger regions of NHPs and people (Fig. 1a-d).

CTD-DSF: Neural population states, trajectories and dynamics. PMd & M1d were among the first cortical areas studied, yet many basic response propertiesremain poorly understood. It remains controversial whether individual-neuron activity relates to muscles or to abstract movement features. Central to this debate is the complex, multi-phasic and heterogeneous individual-neuron responses. One explanation is that responses represent many movement parameters, though numerous studies have shown that this is merely an approximation. We introduced and are advancing an alternate hypothesis, where motor cortex constitutes a dynamical system that in part supports the dynamics themselves which are needed to control movement. This shifts the field from describing individual-neuron responses in somewhat qualitative terms to quantitatively / mechanistically modeling neural-population activity. This is not single-neuron nihilistic: it does not ignore or attempt to average away the complex features of individual-neuron responses. Rather, by capturing the underlying dynamics it is possible to explain the seemingly idiosyncratic responses.

In its simplest, deterministic form 'neural-population state' is governed by a dynamics equation, x(t+1) = f(x(t)) + B u(t), where x(t) is the neural-population state and is a vector describing the firing rate of all neurons at time t. The next time step is x(t+1), f is a nonlinear function, u(t) is an input vector from other brain areas and B is a matrix projecting u(t) into x(t+1). The evolution of x(t) reflects circuit dynamics whose purpose is to produce neural signals for preparing and generating accurate movements. Individual-neuron 'tuning' arises incidentally, and is not elemental. The CTD-DSF predicts that dynamical features and motifs should exist in neural-population responses, which so far has held up across areas, tasks and species.

These nonlinear dynamical systems (NLDS) can often be estimated, but it is typically informative to first assess how a simpler linear dynamical system (LDS) performs. We start with a trained behavior (Fig. 1e), kinematics (Fig. 1f) and action potentials from hundreds of neurons (Fig. 1g-left). After identifying each neuron’s location (Fig. 1g-right, Fig. 1h-left,middle), putative cell-type and directional-tuning preference (Fig. 1h-right), DSF parameter estimation begins.

An LDS is described by a dynamics equation and an observation equation (Fig. 1i). The dynamics equation is x(t+1) = A x(t) + B u(t), where f is replaced by the linear dynamics matrix A. The observation equation, y(t) = C x(t) + d, maps x(t) into the firing rate for each measured neuron (y(t)) via the matrix C and an offset vector d. LDS dynamics can be contractive, expansive, rotational or a fixed point causing a variety of possible neural trajectories (Fig. 1j). Inputs may cause the dynamics (Fig. 1k, gray vectors) to exhibit more complex structure. It is often necessary to engage more complex NLDSs for certain tasks (e.g.,  decision making, full behaviors that transition  between  locally-linear regimes). Recurrent neural networks (RNNs) are a powerful approach for implementing f. We have worked to advance RNNs and their adoption,  including our new "LFADS" technique. Accurately estimating single-trial trajectories is central to understanding the relationship between ms-timescale neural and behavioral events. Illustrating LFADS trajectories, Fig. 1l shows hand trajectories to seven reach targets and Fig. 1m shows reach-direction averaged trajectories inferred  for each  of  44  recording days / sessions projected into  the  3D  subspace that captures the most data variance. These trajectories are highly similar for a given reach direction (color) regardless of the session, providing evidence that the neural circuit dynamics are stable over days/weeks. Fig. 1n shows the single-trial neural population trajectories from within a session day; consistency is evidence of the neural circuit’s well-regulated dynamics.

[1] Yu et al. (2006) NeurIPSChurchland et al. (2006) J NeurosciChurchland et al. (2010) NeuronChurchland*, Yu* et al. (2010) Nat NeurosciChurchland*, Cunningham* et al. (2012) NatureMante*, Sussillo* et al. (2013) NatureKaufman et al. (2014) Nat NeurosciSussillo et al. (2015) Nat NeurosciPandarinath et al. (2018) Nat MethVyas et al. (2018) NeuronVyas et al. (2020) NeuronPeixoto*, Verhein* et al. (2021) Nature.

[2] Santhanam*, Ryu* et al (2006) NatureGilja*, Nuyujukian* et al. (2012) Nat Neurosci; Gilja*, Pandarinath*, et al. (2015) Nat Med; Pandarinath*, Nuyujukian* et al. (2017) eLife; Willett*, Deo* et al. (2020) Cell.


Funding support. We are extremely grateful to the many philanthropists, philanthropies and Federal Agencies -- and to all of the wonderful individual people at these entities that work ceaselessly and serve selflessly for the betterment of human health and society -- that have so generously supported our research through the years and at present. Research is truly a Team effort, including our funding partners. And, finally, we are grateful to literally all US taxpayers who ultimately are the people contributing to support basic and applied science and engineering. This is deeply appreciated. (NIH funding can be found here.)

Arts. Please consider supporting the Arts &


Humanities at the individual, university, city, and federal levels so that they (SHTEAM) may grow together with STEM. Illustration credit iStock / Metamorworks, and licenced by Stanford University.

 

Overview

Last modified: 
Friday, November 26, 2021 - 7:36 am