2.1.

Participants

The study protocol was approved by the Stanford University Institutional Review Board. All participants under 18 years old gave written informed assent, as did their parents or caregivers, and all participants over 18 years old provided written informed consent prior to participation. All subjects received payment at the end of the study for their participation.

To demonstrate the feasibility of the method, four adolescents were recruited to participate in the preliminary study that consisted of five weeks of training intervention. Two subjects had a confirmed diagnosis of ASD and two subjects were TD healthy adolescents. The diagnoses of ASD were confirmed using the Autism Diagnostic Observation Schedule (ADOS-II) Module 3 or 4.³¹ Both children with ASD were “high functioning” with full-scale IQs comparable to children with TD, based on the Wechsler Abbreviated Scale of Intelligence-Second Edition (WASI-II; Wechsler, 2011), and with a parent-reported comprehensive reading level of fourth grade or above. The mental age (chronological $\text{age}\times \text{full scale}$ IQ/100), cognitive skills, and severity of autistic symptoms of the four participants are listed in Table 1.

Table 1

Participants’ characteristics.

2.2.

Experimental Procedure

In each participant’s first visit, several standard neuropsychological and behavioral measures were conducted. The presence of ASD was determined using the ADOS-II, the social responsiveness scale (SRS), and the Social Communication Questionnaire (SCQ). The ADOS-II was administered directly to the participants with ASD by a trained experimenter to ascertain ASD diagnostic status. The SCQ³² and SRS³³ were completed by the parents or designated caregivers of all participants, and these scores were also used to confirm the diagnosis and severity of autistic behavior in participants with ASD. In addition to the parent report and behavioral measures, the complete LFI skills battery evaluation³⁰ was administered to all participants at their first visits. The LFI skills battery is composed of five tests of facial identity, three tests of facial emotion, and two tests of object processing. In this study, we focused on the parts/whole identity test as one of the criteria for individuals with ASD to participate in the study. Children with an ASD diagnosis were eligible for the study if they scored below 85% on this test, indicating deficits in facial-identity recognition. Enrolled participants received training for five sessions, with an average of one training session per week. Participants were semirandomly assigned to receive the training with either real-FB or sham-FB (Table 1). After all training sessions, the LFI skills battery evaluation was administered again to all participants to assess their change in behavioral performance.

Instructions were given to the participants at the beginning of each training session by the experimenter. The instruction was based on a PowerPoint file that explained the task with pictures and simple words. Instructions were consistent for all participants whether they received the real-FB or the sham-FB. In particular, the participants were told that bonus points would be determined by the machine automatically based on their brain activity. To increase their engagement in the task, participants were told that the final score of each training session would be converted to real cash as part of their payment.

2.3.

Task Development

A computer-based task was developed to test and teach the recognition of facial identity utilizing real-time fNIRS neurofeedback. The experiment consisted of a functional localizer task and a facial-identity training task, which ran in sequence. The functional localizer task was used to identify the ROI for each scan. The signal from the determined ROIs was then utilized in the training task to generate the feedback signal in real time. The development of the training task was based on the Face Maker identity subtest of the LFI intervention computer program.²⁹ A 5-s instruction screen was shown right after the functional localizer task to indicate the beginning of the training task. The total duration of the task was about 20 min (task design is shown in Fig. 1).

Fig. 1

Task design.

2.4.

Functional Near-Infrared Spectroscopy Data Acquisition

Functional near-infrared spectroscopy (fNIRS) signal was acquired using an ETG-4000 Optical Topography system (Hitachi Medico Co., Tokyo, Japan) with a sampling rate of 10 Hz. The measurement patches consisted of an evenly distributed array of alternating emitter and detector fiber bundles (optodes). Optodes were spaced 30 mm apart, resulting in a spatial resolution of 30 mm for the system. A channel represented the area measured by one emitter–detector pair, and the channel location was defined as the center position of the emitter–detector optode pairs. A single $3\times 5$ measurement patch containing 15 optodes was positioned over the PFC of each subject’s head, resulting in 22 measurement channels. Another $3\times 5$ measurement patch was positioned over the right temporal cortex (Fig. 2).

Fig. 2

Functional NIRS cap placement and the optodes arrangement. (a) The fNIRS cap placement on a representative participant. (b) The optodes and channel arrangement. (c) A representative projection of channel location on a brain template.

We chose these two regions based on their previously identified roles in cognitive processes involving facial recognition. For individuals with ASD, the face recognition networks usually include aberrant activation in the fusiform face area and significantly increased activation in the PFC, temporal cortex, or primary visual cortex as compared with TD individuals.^36,37 Among these key brain regions, fNIRS can detect only cortical regions underlying the scalp and skull because of the limited penetration depth of the technique. Therefore, we utilized the brain signal measured from the prefrontal and temporal cortices in our study. The individual locations of the optodes on each participant’s head were obtained by a three-dimensional magnetic digitizer system (PATRIOT, Polhemus, Vermont) directly after the task was completed. The technique measured the placement of each optode in relation to five anatomical reference points on the subject’s head (nasion, left and right preauricular points, vertex, and inion).

2.5.

Online Functional Near-Infrared Spectroscopy Data Analysis for Region of Interest and Neurofeedback

The real-time fNIRS signal processing using in-house MATLAB^® software primarily included two functions. The first was to determine the ROI from the functional localizer task and the second was to generate the feedback signal during the training task.

Recorded changes in optical density during the localizer (morph) task were first converted into concentration changes of oxy-Hb ([HbO]) and deoxy-hemoglobin using the modified Beer–Lambert law.³⁸ The channels that contained extremely noisy signals were determined during calibration before the scan began and were excluded from the later analysis. All [HbO] signals were filtered by a bandpass of 0.01–0.5 Hz after the morph task to remove high-frequency instrument noise, physiological heartbeat noise, and low-frequency drift before they were input to a GLM. The GLM analysis was performed online for all [HbO] signals. The standard hemodynamic response function (HRF) and two regressors corresponding to the MF condition and the HP condition, respectively, were used in the analysis. The resulting $t$ values of beta weights for the MF condition were used to determine two primary locations for the fNIRS neurofeedback process: (1) the signal corresponding to the highest $t$ value was selected and the channel location of that signal was considered part of the face-processing network at the temporal cortex, denoted by ${\mathrm{ROI}}^F$; (2) the signal corresponding to the lowest $t$ value was also selected and the channel location of that signal was considered irrelevant to the face-processing network and was used as a reference region, denoted by ${\mathrm{ROI}}^R$.

During the training, the bonus score was calculated in real time directly after the completion of each FM epoch for the real-FB participants. To increase the specificity of the feedback and add an internal control, we fed back the differential fNIRS signals from the two regions of interest (${\mathrm{ROI}}^F$ and ${\mathrm{ROI}}^R$) to the participants. This has the advantage of canceling out any global unspecific effects due to factors such as generally increased blood flow.^39–41 Specifically, we first calculated the percent signal change (PSC) at both ROIs. For instance, the value of the PSC at ${\mathrm{ROI}}^F$ was defined as the difference between the mean value of [HbO] of an FM epoch and the mean value of [HbO] of the paired HM epoch shown right before the FM epoch, divided by the mean value of [HbO] of the HM epoch, i.e., ${\mathrm{PSC}}_i^F=({\overline{[{\mathrm{HbO}}_{\mathrm{FM}}]}}_i-\overline{{[{\mathrm{HbO}}_{\mathrm{HM}}]}_i})/\overline{{[{\mathrm{HbO}}_{\mathrm{HM}}]}_i}$, where $i=\mathrm{1,2},3,\dots ,n$ is the index of the FM epochs (which is the same as the index of the HM epochs). Considering the time lag of the hemodynamic response, we removed the first 5 s of the signal while calculating the mean value for each epoch. Similarly, we calculated the value of PSC (${\mathrm{PSC}}_i^R$) at ${\mathrm{ROI}}^R$. Second, we calculated the difference between the two PSCs, denoted as ${\mathrm{diffPSC}}_i^{\mathrm{FR}}$. When both ${\mathrm{PSC}}_i^F$ and ${\mathrm{diffPSC}}_i^{\mathrm{FR}}$ were larger than zero, 40 bonus points were awarded to the participant; when both ${\mathrm{PSC}}_i^F$ and ${\mathrm{diffPSC}}_i^{\mathrm{FR}}$ were less than zero, zero bonus points were given; for all other conditions, the participant received 20 bonus points. For the sham group, the bonus points were generated for each training session before the task began: a set of numerical data with six zeros, five twenties, and five forties was randomly permutated before each training session to generate a data sequence to be used for the bonus points. At the end of the training task, the participant’s final score was shown on the screen.

2.6.

Outcome Measure

The effect of fNIRS neurofeedback training on facial-identity recognition was measured using the LFI skills battery evaluation tests. The evaluation was administered pre- and post-training sessions. The number of correct responses per minute (CPM) was used as the primary outcome measure to test participants’ improvement in facial recognition performance. The CPM measure was defined as the test score divided by the total reaction time: $\mathrm{CPM}=\text{score}/(\text{sum of reaction time in milliseconds})\times 1000\times 60$. For comparison, we defined the change in CPM as the difference in CPM pre- and post-training sessions, i.e., $\mathrm{\Delta }\mathrm{CPM}={\mathrm{CPM}}_{\mathrm{post}}-{\mathrm{CPM}}_{\mathrm{pre}}$.

2.7.

Offline Data Processing and Analysis

The behavioral learning effect was studied by using the percentage error rate, which was defined as the number of incorrect responses divided by the total number of responses for each training session.

Previous studies show that the posterior superior temporal sulcus (pSTS) is important for face processing.^42–44 For offline analysis of the fNIRS signal, we focused on changes in [HbO] at the pSTS region during the functional localizer task over five training sessions. Specifically, we first generated a mask image of the pSTS in xjview. To determine the accurate location of the pSTS, we utilized the atlas of the posterior portions of the middle and superior temporal cortices and selected a cluster of voxels that were located at the contacting edges of the two regions. All the voxels that had their $y$-axis below $-50$ or above $-20$ in MNI space were removed from the cluster. The remaining voxels were defined as the ROI in the offline analysis and are shown in Fig. 6(c). Considering that the spatial resolution of our fNIRS system was 3 cm, we included all fNIRS channels located within 15 mm of the defined ROI in the following analysis. Specifically, the [HbO] signals from those channels were first filtered by a bandpass of 0.01 to 0.5 Hz. The filtered signals that still had a standard deviation of 0.25 or above were considered extremely noisy based on empirical evidence and were removed from analysis. The remaining signals were averaged and input to the GLM. The contrast values were then calculated between MF and HP conditions for all training sessions and compared among participants.

3.1.

Behavioral Performance

The changes in the number of correct responses per minute ($\mathrm{\Delta }\mathrm{CPM}$) for all four participants are presented in Fig. 3. After five training sessions, all participants were able to improve their performance on the evaluation task. Furthermore, participants with real-FB showed increased improvement in CPM as compared to the participants who received sham-FB.

Fig. 3

Changes in the number of correct responses per minute ($\mathrm{\Delta }\mathrm{CPM}$) for the four participants.

3.2.

Online Data Analysis

Figure 4 shows the ${\mathrm{ROI}}^F$ and ${\mathrm{ROI}}^R$ for the participants with ASD who received real-FB. The red dots (with black circles) represent the corresponding fNIRS ${\mathrm{ROI}}^F$ channel locations and the green dots represent the corresponding fNIRS ${\mathrm{ROI}}^R$ channel locations. The locations of the red dots were scattered, in particular, the red dot of session 3 was located at the PFC.

Fig. 4

The fNIRS channel locations of ${\mathrm{ROI}}^F$ and ${\mathrm{ROI}}^R$ for the participant with ASD who received real-FB at five training sessions are illustrated. The red dots (with black circles) represent the ${\mathrm{ROI}}^F$ channel and the green dots represent the ${\mathrm{ROI}}^R$ channel. As shown in the figure, the positions of the ${\mathrm{ROI}}^F$ are not necessarily consistent across sessions.

3.3.

Offline Data Analysis

Figure 5 shows the percentage error rate for all four participants. The error rate of FM epochs for the participant with ASD who received real-FB [shown as the magenta line in Fig. 5(a)] was decreased by 25% across the five training sessions, while the error rate was decreased by 5% for the participant with ASD who received sham-FB [shown as the magenta line in Fig. 5(b)]. At the fifth training, both TD participants were able to reach an error rate of 2% or below, while both participants with ASD had an error rate of 5% or above. The error rates of HM epochs were comparable for all participants (shown as the black lines in Fig. 5).

Fig. 5

Percentage error rate over the course of five training sessions for (a) the participant with ASD who received real-FB, (b) the participant with ASD who received sham-FB, (c) the TD participant who received real-FB, and (d) the TD participant who received sham-FB.

Figure 6 shows a contrast map of a representative participant and its corresponding fNIRS channel location on the brain template. The map in Fig. 6(b) represents the contrast between the dynamic MF and the static HP conditions during a localizer scan. The blue line in Fig. 6(b) roughly marks the location of the pSTS region that corresponds to the location of the blue line in Fig. 6(a). Notably, the locations of the fNIRS channels were different for each training session, because it was difficult to place the fNIRS optodes in the exact same location for each scan. Thus, the selected channels from ROI region were not the same for each scan session.

The results from the ROI analysis are shown in Fig. 7. The trend lines of the contrasts across the five training sessions for the four participants are also shown. Figure 7(a) shows the change of contrast at the ROI for the two participants with ASD, and Fig. 7(b) shows the change of contrast at the ROI for the two TD participants. The slopes of the trend lines for the two TD participants are both negative and are relatively small compared to the slopes of participants with ASD. The slope of the trend line for the participant with ASD receiving real-FB is also negative, which is similar to the slopes of the TD participants, and the absolute value of the slope was larger than both TDs’, which indicates a trend of normalization of the brain activity. The slope of the trend line for the participant with ASD receiving sham-FB, however, is positive, and the contrast values of the five training sessions are scattered.

Fig. 6

ROI analysis procedure. (a) An fNIRS channel location of a representative participant for a scan session and (b) its corresponding contrast map. (c) The voxel location of the ROI selected for offline analysis.

Fig. 7

Contrast values between MF and the static HP conditions at the ROI across five training sessions for (a) ASD participants and (b) TD participants.