text:
ArticleNephrology Free access | 10.1172/JCI20402
Induction of B7-1 in podocytes is associated with nephrotic syndrome
Jochen Reiser,1 Gero von Gersdorff,1 Martin Loos,1 Jun Oh,1 Katsuhiko Asanuma,1 Laura Giardino,1 Maria Pia Rastaldi,2 Novella Calvaresi,2 Haruko Watanabe,3 Karin Schwarz,1 Christian Faul,1,4 Matthias Kretzler,5 Anne Davidson,1,6 Hikaru Sugimoto,7 Raghu Kalluri,7 Arlene H. Sharpe,8 Jordan A. Kreidberg,9 and Peter Mundel1,4
1Department of Medicine, Albert Einstein College of Medicine, New York, New York, USA. 2Renal Research Association, Renal Immunopathology Center, San Carlo Borromeo Hospital, Milan, Italy. 3Department of Pediatrics, Tokyo Metropolitan Toshima Hospital, Tokyo, Japan. 4Department of Anatomy and Structural Biology, Albert Einstein College of Medicine, New York, New York, USA. 5Department of Medicine, Ludwig-Maximilians-Universität, Munich, Germany. 6Department of Microbiology and Immunology, Albert Einstein College of Medicine, New York, New York, USA. 7Center for Matrix Biology, Department of Medicine, Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, Massachusetts, USA. 8Department of Pathology, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, USA. 9Department of Medicine, Children's Hospital, and Department of Pediatrics, Harvard Medical School, Boston, Massachusetts, USA.
Address correspondence to: Peter Mundel, Division of Nephrology, Albert Einstein College of Medicine, 1300 Morris Park Avenue, New York, New York 10461, USA. Phone: (718) 430-3219; Fax: (718) 430-8963; E-mail: mundel@aecom.yu.edu.
Find articles by Reiser, J. in: JCI | PubMed | Google Scholar
Find articles by von Gersdorff, G. in: JCI | PubMed | Google Scholar
Find articles by Loos, M. in: JCI | PubMed | Google Scholar
Find articles by Oh, J. in: JCI | PubMed | Google Scholar
Find articles by Asanuma, K. in: JCI | PubMed | Google Scholar
Find articles by Giardino, L. in: JCI | PubMed | Google Scholar
Find articles by Rastaldi, M. in: JCI | PubMed | Google Scholar
Find articles by Calvaresi, N. in: JCI | PubMed | Google Scholar
Find articles by Watanabe, H. in: JCI | PubMed | Google Scholar
Find articles by Schwarz, K. in: JCI | PubMed | Google Scholar
Find articles by Faul, C. in: JCI | PubMed | Google Scholar
Find articles by Kretzler, M. in: JCI | PubMed | Google Scholar
Find articles by Davidson, A. in: JCI | PubMed | Google Scholar
Find articles by Sugimoto, H. in: JCI | PubMed | Google Scholar
Find articles by Kalluri, R. in: JCI | PubMed | Google Scholar
Find articles by Sharpe, A. in: JCI | PubMed | Google Scholar
Find articles by Kreidberg, J. in: JCI | PubMed | Google Scholar
Find articles by Mundel, P. in: JCI | PubMed | Google Scholar
First published May 15, 2004 - More info
Published in Volume 113, Issue 10 on May 15, 2004
© 2004 The American Society for Clinical Investigation
First published May 15, 2004 - Version history
Received: October 28, 2003; Accepted: March 23, 2004
Kidney podocytes and their slit diaphragms form the final barrier to urinary protein loss. This explains why podocyte injury is typically associated with nephrotic syndrome. The present study uncovered an unanticipated novel role for costimulatory molecule B7-1 in podocytes as an inducible modifier of glomerular permselectivity. B7-1 in podocytes was found in genetic, drug-induced, immune-mediated, and bacterial toxin–induced experimental kidney diseases with nephrotic syndrome. The clinical significance of our results is underscored by the observation that podocyte expression of B7-1 correlated with the severity of human lupus nephritis. In vivo, exposure to low-dose LPS rapidly upregulates B7-1 in podocytes of WT and SCID mice, leading to nephrotic-range proteinuria. Mice lacking B7-1 are protected from LPS-induced nephrotic syndrome, suggesting a link between podocyte B7-1 expression and proteinuria. LPS signaling through toll-like receptor-4 reorganized the podocyte actin cytoskeleton in vitro, and activation of B7-1 in cultured podocytes led to reorganization of vital slit diaphragm proteins. In summary, upregulation of B7-1 in podocytes may contribute to the pathogenesis of proteinuria by disrupting the glomerular filter and provides a novel molecular target to tackle proteinuric kidney diseases. Our findings suggest a novel function for B7-1 in danger signaling by nonimmune cells.
The kidney glomerulus is a highly specialized structure that ensures selective ultrafiltration of plasma so that essential proteins are retained in the blood. The common denominator in a variety of kidney diseases, such as diabetes mellitus, focal segmental glomerulosclerosis, and systemic lupus erythematosus, is glomerular dysfunction (nephrotic syndrome), involving a massive loss of protein in the urine (proteinuria) (1). Kidney or systemic diseases that result in nephrotic syndrome are important causes of morbidity and mortality affecting both adults and children. Glomerular podocytes serve as the final barrier to urinary protein loss by the formation and maintenance of podocyte foot processes (FPs) and the interposed slit diaphragm (SD) (1). All forms of nephrotic syndrome are characterized by abnormalities in podocytes, including retraction (effacement) of podocyte FPs and/or molecular reorganization of the SD. FP effacement is a dynamic process that correlates closely with the development of proteinuria both in human disease and in experimental models. FP effacement requires a precise interplay of multiple cellular functions, including structural alterations of the cytoskeleton, movement of FPs over the basement membrane, and reconstruction of the SD (1). From a clinical perspective, it is important to recognize that early structural changes in podocytes such as substructural alterations in SD composition or FP effacement have to be reversed within a certain period of time to prevent development of severe and progressive glomerular damage (2). Recent years have been extremely fruitful in providing an extensive amount of information on the molecular composition of the SD and its anchoring structures, which form multiple interactions and provide signaling platforms. The discovery of several novel podocyte proteins and their mutation analysis, including nephrin (3), CD2AP (4), α-actinin-4 (5), podocin (6), and neph1 (7), have shed light on the pathogenesis of proteinuria and emphasized the critical role of the podocyte and the SD in maintaining the function of the glomerular filtration barrier.
Podocyte FPs form a scaffolding around the capillary loops that are anchored to the glomerular basement membrane (GBM) via α3β1 integrin (8) and α-/β-dystroglycans (9, 10). α3β1 integrin represents a receptor for certain isoforms of laminin, including laminin-5 in the epidermal basement membrane and laminin-10/11 in the GBM (11). In podocytes, α3β1 integrin is expressed at the basal side of FPs. Podocytes of α3 integrin–deficient (α3–/–) mice bear a strong morphological resemblance to podocytes in congenital nephrotic syndrome and other pathological conditions with FP effacement (11, 12). The α3 integrin deletion therefore represents a genetic model system that allows the analysis of the molecular basis of FP assembly and dynamics.
B7-1, also termed CD80, is a transmembrane protein normally expressed on the surface of B cells and other APCs (13–15). On APCs, B7-1 acts as a costimulatory molecule through binding to its cognate receptors CD28 and CTLA-4 on T cells (13–15). Little is known about B7-1 expression by non–bone marrow–derived cells. In human keratinocytes, B7-1 gene expression is upregulated by allergens and irritants (16). In P815 mastocytoma tumor cells, B7-1 gene expression is induced by oxidative stress (17), which is reminiscent of the induction of B7-1 in podocytes by puromycin, described below. In the kidney, B7-1 can be induced in glomerular endothelial cells by warm ischemia/reperfusion injury in rats (18).
The innate immune system senses pathogen-associated molecular motifs through toll-like receptors (TLRs), thereby initiating an immunological response (19). In APCs, LPS is the strongest endogenous inducer of B7-1 expression through TLR-4 signaling (20). Here we show that danger signaling occurs also in non–bone marrow–derived glomerular podocytes and underlies a dramatic physiological response. Podocytes sense LPS by TLR-4, which triggers the reorganization of the kidney-filtration apparatus, leading to podocyte FP effacement and proteinuria. This response requires the induction of costimulatory molecule B7-1, since proteinuria was abrogated in B7-1–/– mice. We discuss a novel function for B7-1 in the danger signaling of nonimmune cells.
Cell culture. WT podocytes were cultured as described before (21). We generated conditionally immortalized α3–/– podocyte cell lines by intercrossing α3 integrin heterozygous mice (12) with the ImmortoMouse (Charles River Laboratories Inc., Wilmington, Massachusetts, USA), which carries a temperature-sensitive T antigen as transgene (22). Glomeruli were isolated from kidneys of E18 homozygous embryos, and eight α3–/– clonal podocyte cell lines were obtained by limiting dilution. To show that observed changes in podocyte gene expression and function were due to the absence of α3 integrin, three α3–/– clones were stably transfected with human α3 integrin cDNA as described before for α3–/– collecting duct epithelial cells (23).
Differential display PCR. Total RNA was isolated from differentiated WT and α3–/– podocytes that had been grown under nonpermissive conditions for 14 days using the RNeasy kit (QIAGEN Inc., Valencia, California, USA). After DNase treatment, reverse transcription was carried out with three different anchored synthetic oligonucleotide primers from the RNAimage kit (GenHunter, Nashville, Tennessee, USA). Differential display PCR reactions were performed in duplicate using 96 different primer combinations exactly as described before (24). Differentially expressed PCR products were cloned into the pCR2 vector using the TA cloning kit (Invitrogen, Carlsbad, California, USA) and sequenced. Sequence analyses and database searches were done with the software program package HUSAR (Heidelberg Unix Sequence Analysis Resources; German Cancer Research Center, Heidelberg, Germany). The following primers specific for B7-1, yielding a 172-bp product, were used: sense, 5′-TGTATGCCCAGGAAACAGGT-3′; antisense, 5′-AGCCCGATCACCAC-TGATTA-3′.
Antibodies. The following protein-specific primary antibodies were used: anti–B7-1 (16-10A1; Pharmingen, San Diego, California, USA), anti–B7-1 (R&D Systems Inc., Minneapolis, Minnesota, USA), and mouse monoclonal anti-synaptopodin (25). Rabbit was a gift from L. Holzman (University of Michigan, Ann Arbor, Michigan, USA); rabbit polyclonal anti-nephrin CD2AP was a gift from A. Shaw (Washington University, St. Louis, Missouri, USA); polyclonal anti-ZO-1 was from Zymed Laboratories Inc. (South San Francisco, California, USA).
Immunocytochemistry. Human and murine kidneys were harvested and snap-frozen according to standard protocols and fixed after sectioning in ice-cold acetone for 10 minutes. For immunofluorescent labeling, sections were washed once with PBS and incubated with blocking solution (2% FCS, 2% BSA, 0.2% fish gelatin) for 30 minutes at room temperature before further incubation with one of the primary antibodies for 1 hour at room temperature. For double labeling, sections were washed three times with PBS for 5 minutes, and the second primary antibody was applied for 60 minutes. Antigen-antibody complexes were visualized with secondary antibodies (Rockland Inc., Gilbertsville, Pennsylvania, USA) conjugated with fluorochrome (Cy2 or Cy3). Pictures were captured with a Spot cooled CCD camera (Diagnostic Instruments, Sterling Heights, Michigan, USA) and processed with Adobe Photoshop 5.0 software (Adobe Systems Inc., San Jose, California, USA). Confocal microscopy and processing of micrographs were performed as reported before (25).
LPS treatment of cultured podocytes. Differentiated podocytes were exposed to different doses of LPS (1–50 ∝g/ml) for 3.5 to 72 hours. RT-PCR for B7-1 was done as described above, and changes in the organization of the actin cytoskeleton were assessed by phalloidin staining as described before (26). Each set of experiments was repeated at least five times.
Nephrin–/– mice. The generation and functional characterization of nephrin–/– mice has been reported before (27). For the present study, kidneys were harvested from 1-day-old nephrin–/– and WT littermate animals, snap-frozen in liquid nitrogen, and processed for immunohistochemistry as described above.
LPS-induced FP effacement and proteinuria. All animal studies were approved by the Albert Einstein College of Medicine Animal Institute Committee. B7-1–/– mice were generated and backcrossed onto a BALB/c background (>12 generations) as previously described (28). SCID mice (29) on a BALB/c background as well as WT 129 and C57BL/6 mice were obtained from The Jackson Laboratory (Bar Harbor, Maine, USA). The base-line urine protein excretion of 4- to 6-week-old female 129 (n = 10), C57BL/6 (n = 25), BALB/c (n = 13), B7-1–/– (n = 5), and SCID (n = 6) mice was analyzed by urine dipstick (Albustix; Bayer, Tarrytown, New York, USA) before animals were injected intraperitoneally with either 200 ∝g LPS (1 mg/ml in sterile LPS-free PBS) in a total volume of 200 ∝l, or equal volumes of sterile LPS-free PBS (PBS: BALB/c WT, n = 5; B7-1–/–, n = 3; SCID, n = 3). Twenty-four hours after injection, urinary protein excretion was measured again, and kidneys were harvested and processed for H&E and immunofluorescence staining. FP effacement was assessed by transmission electron microscopy according to our published protocols (30). In a subset of C57BL mice (n = 5), urine was collected before and 24 hours after LPS injection and analyzed by dipstick. From the same urines we determined the protein/creatinine ratio with Albuwell M and Creatinine Companion protein/creatinine ELISA kits (Exocell Inc., Philadelphia, Pennsylvania, USA), respectively. All measurements were performed in duplicate. The following results were obtained: protein/creatinine ratio: control, 26 ± 12.78; 24 hours' LPS, 205 ± 150.01, P < 0.01; dipstick: control (trace = 0), 0.0 ± 0.21; 24 hours' LPS, 2.0 ± 0.71, P < 0.001. These results showed that changes in urinary protein detected with both tests were comparable.
Microbead experiments. The coating of microbeads was adapted from the method of Sedwick and coworkers (31). Sulfate-coated blue fluorescent microbeads (6.2 ∝m in diameter; Interfacial Dynamics Corp., Portland, Oregon, USA) were coated with polyclonal anti–B7-1 (16-10A1) antibody or polyclonal anti-dendrin (negative control). In addition, murine CTLA-4-Ig with a mutated Fc-binding site (32) or uncoated microbeads were used. Twenty microliters of beads per 5 ∝g protein were used for surface coating. Podocytes growing in 24-well plates were incubated overnight with the coated beads, followed by immunocytochemistry.
B7-1 is induced in podocytes in genetic and drug-induced models of FP effacement. Podocytes in kidneys of α3–/– mice are unable to maintain normal podocyte structure, including the elaboration of mature FPs along the GBM (12). Therefore α3–/– mice provide an excellent genetic model to study podocyte FP effacement. α3–/– mice die during the perinatal period with severe kidney and lung defects, precluding extensive study of the role of this cell matrix receptor in podocytes in vivo. To delineate the molecular mechanisms underlying FP effacement in α3–/– mice, we took advantage of our previously established cell culture system for WT podocytes (21) and generated α3–/– conditionally immortalized podocyte cell lines. In an effort to identify genes critically involved in the development of FP effacement and proteinuria, we performed differential display PCR of total RNA harvested from cultured WT and α3–/– podocytes. Three PCR products (A31–3) were differentially expressed between WT and α3–/– podocytes. Two clones, A31 (Figure 1A) and A32, were expressed by α3–/– cells and one, A33, only by WT cells (data not shown). We confirmed the differential expression of all three clones by quantitative RT-PCR (24) (Figure 1B and data not shown). The expression of A31 was 14-fold higher in α3–/– than in WT podocytes (Figure 1B) and therefore was evaluated in detail. The 400-bp product of A31 was identical with a mouse myotube EST clone (AA673837). Using the 400-bp product as a probe, we isolated one clone from a mouse skeletal muscle cDNA library that covered the complete EST clone and extended 516 bp into exon 6 of the mouse B7-1 gene (33). The overlap with B7-1 ended at the 3′ end of exon 6. Downstream from exon 6 was complete sequence diversion from the previously identified mouse B7-1 cDNA, suggesting the existence of an additional exon 7 forming a portion of the 3′ untranslated region (EMBL/GenBank accession no. AJ278965). The same novel splice variant of B7-1 was also cloned by RT-PCR from α3–/– podocytes (data not shown). We then tested whether the restoration of α3 integrin expression would diminish B7-1 expression. Stable transfection of α3–/– podocytes with human α3 integrin cDNA returned B7-1 expression to WT levels (Figure 1C). A control RT-PCR for the housekeeping gene GAPDH showed equal RNA concentrations (Figure 1C). Similar to the inactivation of the α3 integrin gene, a single injection of puromycin aminonucleoside (PAN) induces podocyte FP effacement and proteinuria in rats (34) and mice (35). In vitro, puromycin leads to the reorganization of the actin cytoskeleton, focal contacts, and SD proteins in podocytes (26, 36). The application of puromycin to WT podocytes for 48 hours caused a dose-dependent increase in B7-1 mRNA levels (Figure 1D). To test whether the effect of puromycin on B7-1 induction in WT cells was independent of α3 integrin expression, the mRNA expression of B7-1 and α3 integrin was analyzed in WT podocytes exposed to PAN for 6 and 24 hours. An induction of B7-1 mRNA expression could be observed, whereas the α3 integrin mRNA expression was not altered (Figure 1E). These studies showed that the induction of B7-1 can occur independent of changes in α3 integrin expression. To explore whether B7-1 was induced in α3–/– podocytes in situ, we analyzed the expression of B7-1 in kidney sections of E18 embryonic WT and α3–/– mice by immunofluorescence microscopy. No B7-1 labeling was found in WT kidneys (Figure 1F, left panel). In contrast, a strong expression of B7-1 was found in podocytes of α3–/– kidneys as shown by double labeling with the podocyte marker synaptopodin (25) (Figure 1F, right panels). In summary, the induction of B7-1 in both genetic and drug-induced models suggests that podocyte B7-1 expression can be regulated by various stimuli and may play a general role in different diseases with FP effacement and proteinuria.
B7-1 expression in podocytes. (A) Differential display PCR of WT (+/+) and α3_/_ (_/_) podocytes. The arrow indicates the position of B7-1. (B) Induction of B7-1 mRNA expression in α3_/_ podocytes. RT-PCR from three independent samples (a_c) on serial dilution (1:5 to 1:25) of cDNA standardized for β-actin. Con, H2O control. (C) Rescue (Res) of α3_/_ podocytes by stable transfection with human α3 integrin restored B7-1 expression to WT levels. RT-PCR for GAPDH shows equal RNA concentrations. (D) Dose-dependent increase of B7-1 mRNA in WT podocytes in response to puromycin (0.1, 1, 5, 10, and 50 ∝g/ml). (E) Time course of B7-1 and α3 integrin mRNA expression in puromycin aminonucleoside_treated (PAN-treated) WT podocytes. As early as 6 and 24 hours, B7-1 is induced by PAN, whereas α3 integrin levels remain unchanged. (F) B7-1 is not expressed in E18 WT kidneys (left panel) but is expressed in podocytes of α3_/_ kidneys as shown by double labeling with synaptopodin (right panels).
Podocyte B7-1 expressed in human and murine lupus nephritis correlates with disease stage and severity of proteinuria. To explore the significance of B7-1 in podocytes in renal diseases with FP effacement and urinary protein loss, we analyzed B7-1 expression in New Zealand black/white (NZB/W) F1 mice, an established murine model of lupus glomerulonephritis (37). While no B7-1 expression was detected in WT kidneys (Figure 2A, left panel), a strong induction was noted in glomeruli of mice with active lupus glomerulonephritis (Figure 2A, middle panel). At higher magnification, B7-1 was detected in podocytes overlying glomerular capillaries (Figure 2A, right panel). In addition, as expected, B7-1 expression was found in interstitial lymphocytes (Figure 2A, middle panel, arrows). Double-labeling studies with synaptopodin (25) showed that glomerular B7-1 was expressed only in podocytes, not in mesangial cells or infiltrating cells (Figure 2B). Importantly, the expression of B7-1 in podocytes correlated with the severity of proteinuria, since B7-1 was strongly expressed in 30-week-old 4+ proteinuric lupus mice (Figure 2B) but not in age-matched animals that showed only trace proteinuria (Figure 2C). To determine the significance of glomerular B7-1 expression in human glomerulonephritis, we analyzed B7-1 expression in human kidney biopsies from different stages of lupus nephritis. Glomerular B7-1 expression correlated with the disease stage in that it increased from WHO class II to V and was strongest in WHO class V kidney biopsies (Figure 2, D and E). As in the murine condition (Figure 2C), glomerular B7-1 colocalized with the podocyte marker synaptopodin (25) (Figure 2E). Taken together, these data underscore the clinical relevance of B7-1 expression in podocytes. They suggest that B7-1 in podocytes contributes to the pathogenesis of proteinuria.
B7-1 expression in podocytes in murine and human lupus nephritis correlates with the severity of proteinuria. (A) B7-1 is not expressed in normal mouse kidney (left). In the lupus kidney, strong glomerular (G) B7-1 expression is observed (middle). In addition, infiltrating interstitial B cells stain positive (arrows). At higher magnification (right), B7-1 expression is found in podocytes (arrows) in the glomerulus. (B) Double labeling with synaptopodin confirmed that intraglomerular B7-1 expression is restricted to podocytes in 4+ proteinuric (4+ prot) nephritis. (C) B7-1 is absent in an age-matched lupus mouse with trace proteinuria (trace prot). (D and E) In human lupus nephritis, glomerular B7-1 expression increases from WHO class II to V and colocalizes with synaptopodin in podocytes (E).
Induction of B7-1 in podocytes of nephrin–/– mice. The nephrin gene NPHS1 is the target gene of congenital nephrotic syndrome of Finnish type, an autosomal-recessive disorder, characterized by massive proteinuria in utero and nephrosis at birth (3). Inactivation of NPHS1 in mice results in massive proteinuria, edema, and perinatal lethality (27, 38). To test whether the induction of B7-1 in podocytes under nephrotic conditions is specific for lupus nephritis, we analyzed the expression of B7-1 in nephrin–/– mice (27). B7-1 colocalized with synaptopodin in podocytes of nephrin–/– mice (Figure 3A). In contrast, no podocyte B7-1 expression was found in WT littermates (Figure 3B). These data imply that podocyte B7-1 expression is not specific for lupus nephritis but is also found in noninflammatory conditions with proteinuria and is not dependent on immune complexes.
Induction of B7-1 in podocytes of nephrin_/_ mice. (A) B7-1 is induced in podocytes of nephrin_/_ mice as shown by double labeling with synaptopodin. (B) B7-1 is absent from podocytes of WT littermate controls. These data establish that the expression of B7-1 in podocytes in vivo can occur independently of immune complexes or inflammatory cells.
B7-1 expression in podocytes in vitro is induced by LPS through TLR-4 signaling and leads to the reorganization of the actin cytoskeleton. B7-1 expression in APCs can be induced by LPS through TLR-4/CD14–mediated signaling (20). Interestingly, podocytes constitutively express TLR-4 and its coreceptor CD14 and hence respond to LPS (Figure 4A). Therefore, we explored the effects of LPS on podocytes. Exposure to LPS specifically upregulated B7-1 (Figure 4B) and induced the disruption of stress fibers (Figure 4C), creating a cortical ring of F-actin and a submembranous space lacking actin fibers (Figure 4D). This cytopathic effect is reminiscent of the actin changes that are observed in FP effacement in vivo (1) and suggests that LPS may contribute to the pathogenesis of FP effacement and proteinuria through the reorganization of the podocyte actin cytoskeleton.
LPS-induced upregulation of B7-1 and actin cytoskeleton reorganization in podocytes in vitro. (A) Cultured podocytes express the LPS receptors CD14 and TLR-4 as shown by RT-PCR. (B) LPS treatment upregulates B7-1 mRNA expression and reorganizes the podocyte cytoskeleton from a stress fiber pattern (C), creating a cortical ring of F-actin (D; arrowheads) and a submembranous space devoid of actin fibers (asterisks).
LPS upregulates B7-1 in podocytes in vivo and leads to nephrotic syndrome in WT but not in B7-1–/– mice. To test this hypothesis further, we injected a single low dose of LPS intraperitoneally into adult female WT BALB/c mice and analyzed the effects on kidney histology, podocyte B7-1 expression, and urinary protein excretion. Histologically, kidneys of LPS-injected mice were not distinguishable from PBS-injected controls (Figure 5A). In particular, no signs of inflammation were found. LPS induced upregulation of B7-1 in podocytes (Figure 5B) and FP effacement within 24 hours (Figure 5D, left panel). In contrast, PBS injection did not induce podocyte B7-1 expression (Figure 5C) or FP effacement (Figure 5D, right panel). Most significantly, LPS injection caused severe proteinuria (n = 13; base line, 0.8 ± 0.4, vs. LPS, 2.7 ± 0.9; P < 0.001, two-sided t test) within 24 hours (Figure 5G). In contrast, no proteinuria was found in PBS-injected controls (data not shown). We next established that LPS-induced nephrotic syndrome is independent of the mouse strain, since we found it also in WT 129 (n = 9; base line, 0.7 ± 0.5, vs. LPS, 2.1 ± 0.78; P < 0.001, t test) and C57BL/6 mice (n = 23; base line, 0.5 ± 0.29, vs. LPS, 1.9 ± 0.56; P < 0.0001, t test) (Figure 5G). The LPS-induced proteinuria was transient and returned to base-line levels after 72 hours (data not shown). To establish a causal relationship between podocyte B7-1 upregulation and development of proteinuria, the experiments were repeated with B7-1–/– mice (28). No proteinuria was observed in LPS-injected B7-1–/– mice (n = 5; base line, 1.0 ± 0, vs. LPS, 0.6 ± 0.55; P not significant, t test) (Figure 5G). The differences in proteinuria between WT and B7-1–/– mice were statistically significant (P < 0.001, t test). These data suggest that B7-1 induction in podocytes is necessary for the development of LPS-induced proteinuria.
LPS-induced FP effacement and proteinuria are B7-1 dependent. (A) LPS injection (left panel) and PBS injection (right panel) do not cause renal inflammation 24 hours after injection in WT mice. (B) LPS injection induces B7-1 in podocytes as shown by double labeling with synaptopodin. (C) PBS-injected control mice do not upregulate B7-1. (D) LPS injection causes severe FP effacement within 24 hours in WT mice (left panel; arrows) but not in PBS-injected injected control animals (right panel). (E and F) B7-1 is upregulated in podocytes of LPS-treated SCID mice (E), but not in PBS-injected SCID controls (F). (G) LPS injection causes severe proteinuria in 129, C57BL/6, and BALB/c WT mice and SCID mice but not in B7-1_/_ mice. Asterisks indicate statistically significant changes (see text for details).
LPS-induced nephrotic syndrome is independent of T or B cells. To rule out that the LPS-induced proteinuria resulted from the activation of T or B cells rather than from a direct action on podocytes, SCID mice, which are devoid of T and B cells (29) were treated with LPS. As in WT mice, LPS injection in SCID mice caused podocyte B7-1 expression (Figure 5E) and severe proteinuria (n = 6; base line, 1.2 ± 0.4, vs. LPS, 3.2 ± 0.9; P < 0.001, t test) within 24 hours (Figure 5G). In contrast, B7-1 staining was absent from PBS-treated SCID mice (Figure 5F). Hence, the induction of B7-1 expression by podocytes in response to LPS is independent of T or B cells. Taken together, these data indicate a causal link between podocyte B7-1 expression and urinary protein loss that is independent of lymphocyte infiltration or activation.
B7-1 modifies the glomerular filter by sequestration of SD proteins. How does B7-1 upregulation in podocytes contribute to the pathogenesis of proteinuria? To test whether the ligation of B7-1 transduces intracellular signals in podocytes, thereby modulating the composition of the SD, we used a microbead approach to cross-link B7-1. With this approach, three key proteins of the SD, nephrin (3), CD2AP (4), and ZO-1 (39), were analyzed in α3–/– podocytes by immunocytochemistry after binding of microbeads coated with CTLA-4-Ig or anti–B7-1 antibody (Figure 6). The detailed microscopic analysis revealed that nephrin was redistributed from cell-cell contacts (Figure 6A) and accumulated around CTLA-4-Ig–coated (Figure 6B) or anti–B7-1–coated microbeads (Figure 6C). The association of SD proteins with anti–B7-1–coated microbeads was confirmed by confocal microscopy, which showed a ring of nephrin lining the microbeads (Figure 6D). Similar results were obtained for CD2AP, which was redistributed from the cell membrane and the actin filaments (Figure 6E) to microbeads (Figure 6, F–H). ZO-1 relocated from cell-cell junctions (Figure 6I) to the microbeads (Figure 6, J–L). To show that the redistribution of SD proteins was specifically mediated by B7-1, several control experiments were performed. First, microbeads were coated with a mutated variant of murine CTLA-4-Ig that can no longer bind to Fc receptor. Second, microbeads were coated with an irrelevant antibody with an intact Fc fragment. The microbeads coated with the irrelevant antibody (dendrin) could to some extent adhere to podocytes but did not cause the rearrangement of ZO-1 (Figure 6M). Third, uncoated microbeads were used. No binding was observed with uncoated microbeads or beads coated with the irrelevant antibody (data not shown). Moreover, B7-1 clustering beads had no effect on cultured WT podocytes, which do not express B7-1 (data not shown). In summary, the results of the bead experiments are specific and conclusive. They suggest that B7-1 may contribute to the pathogenesis of proteinuria by disrupting the SD complex.
Redistribution of SD proteins to sites of B7-1 engagement. (A) In untreated α3_/_ podocytes, nephrin is expressed at cell-cell contacts. (B and C) Incubation of podocytes with microbeads coated with CTLA-Ig (B) or anti_B7-1 antibody (C) redistributes nephrin around microbeads (arrows). (D) The association of nephrin with microbeads was confirmed by confocal microscopy. (E_L) Similar redistribution patterns were seen for CD2AP (E_H) and ZO-1 (I_L). Arrows in F and J: accumulation of CD2AP (F) and ZO-1 (J) around microbeads. Asterisk in K: microbead. (M) In contrast, microbeads coated with an irrelevant antibody could to some extent adhere to podocytes (arrowhead) but did not affect the localization of ZO-1 at cell-cell contacts (arrows).
This study is the first to demonstrate that B7-1 is expressed in podocytes under a variety of stress conditions. Moreover, the results of this study establish the clinical significance of podocyte B7-1 expression and elucidate its role in the pathogenesis of nephrotic syndrome. We specifically demonstrate that podocyte B7-1 reorganizes the actin cytoskeleton of podocytes and modulates SD organization. The effects are independent of T cells and B cells and represent a novel unique function of B7-1. This study also is the first to show that LPS injection in mice induces B7-1–dependent nephrotic syndrome with FP effacement and urinary protein loss.
So far, B7-1 had been assumed to be predominantly expressed on B cells and other professional APCs. The traditional role of B7-1 is T cell costimulation (13–15). Along with the interaction of the T cell receptor with antigen-MHC complexes, the interaction of B7 with its receptors CD28 and CTLA-4 on T cells plays a pivotal role in T cell–mediated immunity (13–15). In T cells, binding of B7-1 to its receptors on the T cell surface triggers migration of protein kinases, actin-binding proteins, and other accessory molecules to the T cell–APC interface (40, 41). This formation of the immunological synapse causes a rearrangement of the T cell actin cytoskeleton (40, 42–44), activates protein tyrosine kinases that couple to PI3K, and controls the activity of Ras- and Rho-family GTPases in T cells (45).
The role of B7-1 in T cell costimulation is well established and defines a dialogue between a B7-1–expressing APC and a T cell to which the costimulatory signal is conveyed (13–15). In contrast, B7-1–mediated outside-in signaling in B7-1–expressing APCs is still obscure. With respect to outside-in signaling through B7-1 on B7-1–expressing cells, two studies have been published. One study showed that cross-linking of B7-1 on B lymphoma Raji cells induces tyrosine phosphorylation of intracellular proteins, indicating a role of B7-1 in outside-in signaling (46). Further support for a role of B7-1 in outside-in signaling comes from a recent report showing that CTLA-4-Ig–mediated ligation of B7-1 regulates tryptophan catabolism in dendritic cells (47). These changes are reminiscent of the changes seen in SD protein redistribution after ligation/activation of B7-1 in podocytes (Figure 6).
In human keratinocytes, B7-1 gene expression is induced by allergens and irritants (16). In P815 mastocytoma tumor cells, B7-1 gene expression is induced by oxidative stress (17). In APCs, MyD88-independent LPS/TLR-4 signaling is the strongest activator of B7-1 expression (20, 48). Similarly, B7-1 induction in podocytes appears to be downstream of various signaling pathways, including TLR signaling (Figure 5). Generally, somatic cells grown in culture are, for the most part, quite indifferent to LPS (20), and earlier studies had shown that macrophages are the primary target cells of LPS in septic shock (20). Here we demonstrate that podocytes constitutively express the LPS receptor TLR-4 and its coreceptor CD14 and respond to LPS exposure with upregulation of B7-1. This is another unanticipated finding of the present study.
The LPS experiments (Figures 4 and 5) allowed us to explore several important questions. First, they established that the induction of podocyte B7-1 in conditions with FP effacement and proteinuria was causative and not the consequence of the nephrotic condition, since B7-1–/– mice were protected from LPS-induced proteinuria. Second, one might argue that the effects of LPS are predominantly mediated through the action of T cells. Our data refute this hypothesis, because (a) SCID mice lacking T and B cells showed the same degree of FP effacement and proteinuria as WT mice; (b) exposure of cultured podocytes in the absence of any other cells was sufficient to upregulate B7-1 and reorganize the podocyte actin cytoskeleton; and (c) the cell-free microbead-mediated cross-linking of B7-1 on α3–/– podocytes was sufficient to redistribute vital SD proteins. Collectively, these data suggest that B7-1 expression by podocytes represents a pathogenic mechanism for disruption of the glomerular filtration apparatus that is independent of T cell activation.
An intriguing question remains, why do podocytes upregulate B7-1? In the grass shrimp, Palaemonetes pugio, podocytes are involved in the ultrafiltration of hemolymph where micro- and macromolecular substances are cleared (49). In the pre-molt stage of this shrimp, podocyte assemblages extend into the efferent hemolymph channels and are in direct contact with immune cells, and they may serve as a component of the LPS-sensing (20) innate immune response. The upregulation of podocyte B7-1 by LPS may be a phylogenetically preserved mechanism, which can be reactivated under pathological conditions, e.g., in patients with Gram-negative sepsis. Consistent with this idea, the development of transient proteinuria has been found during the course of Gram-negative sepsis (50). Moreover, a prospective study showed that in postoperative septic patients microalbuminuria is an early indicator of increased glomerular permeability, but not in nonseptic patients (51). The resulting, transient, proteinuria may be a physiological response aimed at clearing the circulation from pathogen-associated molecules. In human minimal-change disease, the onset of nephrotic syndrome is often preceded by an infection or allergic reaction (52). This raises the intriguing possibility that nephrotic syndrome in minimal-change disease represents the persistence of a normally beneficial response caused by genetic defects in the B7-1 pathway or by other unphysiological activators. The sustained B7-1 activation in podocytes, in turn, results in continuing nephrotic syndrome.
In summary, we have demonstrated a novel function for B7-1 in the pathogenesis of proteinuria that is distinct from its role in costimulation (Figure 7). Our findings suggest a novel function for B7-1 in danger signaling by podocytes. The B7-1 knockout by itself is not conclusive since it is not podocyte specific. While the SCID mice control for T or B cell–mediated effects, it is still possible that dendritic cells or other APCs might contribute to the observed effects. Future studies with a podocyte-specific B7-1 knockout should allow us to address this question.
Pathways leading to B7-1_mediated proteinuria. Various stimuli may lead to podocyte B7-1 induction. These include genetic stimuli (e.g., deletion of α3 integrin or nephrin), toxic stimuli (PAN-induced reactive oxygen species [ROS]), or direct stimulation of the TLR-4/CD14 receptor on the podocyte. B7-1 then induces FP effacement and disruption of the SD complex, thereby modifying glomerular permselectivity.
We propose that transient B7-1–induced nephrotic syndrome may be a physiological response. Clearly, future studies will be required to explore this hypothesis in detail, but it could potentially explain how a normally beneficial danger-sensing mechanism can have deleterious effects in certain genetic backgrounds. Understanding the role of B7-1 in podocytes should allow new avenues toward the development of novel, selective podocyte-protective therapies that modulate danger signaling in podocytes and tackle proteinuria. It will be interesting to see whether B7-1–dependent pathways are also operative in chronic kidney disease.
We thank Torsten Heider for technical assistance and Michael Cammer (Albert Einstein College of Medicine) for help with confocal microscopy. We thank Andrey S. Shaw for anti-CD2AP and Lawrence B. Holzman for anti-nephrin antibodies and Amin Arnaout for helpful discussion and critical reading of the manuscript. Jochen Reiser was a fellow of the Graduiertenkolleg Experimentelle Nieren- und Kreislaufforschung of the Deutsche Forschungsgemeinschaft (DFG) and Martin Loos of the Deutscher Akademischer Austauschdienst. Gero von Gersdorff is supported by a fellowship from the Lupus Foundation of New York, and Karin Schwarz and Katsuhiko Asanuma by the Kidney and Urology Foundation of America (KUFA). Jun Oh was supported by the DFG and by KUFA. This work was supported by grants from the Lupus Research Institute, the NIH (DK57683 and DK062472), and the George M. O'Brien Kidney Center (DK064236) to Peter Mundel, as well as NIH grant DK55001 to Raghu Kalluri.
Jochen Reiser's present address is: Renal Unit, Massachusetts General Hospital, and Harvard Medical School, Charlestown, Massachusetts, USA.
Nonstandard abbreviations used: foot process (FP); glomerular basement membrane (GBM); puromycin aminonucleoside (PAN); slit diaphragm (SD); toll-like receptor (TLR).
Somlo, S, Mundel, P. Getting a foothold in nephrotic syndrome. Nat. Genet. 2000. 24:333-335.
Kriz, W, Gretz, N, Lemley, KV. Progression of glomerular diseases: is the podocyte the culprit? Kidney Int. 1998. 54:687-697.
Kestila, M, et al. Positionally cloned gene for a novel glomerular protein—nephrin—is mutated in congenital nephrotic syndrome. Mol. Cell. 1998. 1:575-582.
Shih, NY, et al. Congenital nephrotic syndrome in mice lacking CD2-associated protein. Science. 1999. 286:312-315.
Kaplan, JM, et al. Mutations in ACTN4, encoding alpha-actinin-4, cause familial focal segmental glomerulosclerosis. Nat. Genet. 2000. 24:251-256.
Boute, N, et al. NPHS2, encoding the glomerular protein podocin, is mutated in autosomal recessive steroid-resistant nephrotic syndrome. Nat. Genet. 2000. 24:349-354.
Donoviel, DB, et al. Proteinuria and perinatal lethality in mice lacking neph1, a novel protein with homology to nephrin. Mol. Cell. Biol. 2001. 21:4829-4836.
Adler, S. Characterization of glomerular epithelial cell matrix receptors. Am. J. Pathol. 1992. 141:571-578.
Regele, HM, et al. Glomerular expression of dystroglycans is reduced in minimal change nephrosis but not in focal segmental glomerulosclerosis. J. Am. Soc. Nephrol. 2000. 11:403-412.
Raats, CJ, et al. Expression of agrin, dystroglycan, and utrophin in normal renal tissue and in experimental glomerulopathies. Am. J. Pathol. 2000. 156:1749-1765.
Kreidberg, JA. Functions of alpha3beta1 integrin. Curr. Opin. Cell Biol. 2000. 12:548-553.
Kreidberg, JA, et al. Alpha 3 beta 1 integrin has a crucial role in kidney and lung organogenesis. Development. 1996. 122:3537-3547.
Abbas, AK, Sharpe, AH. T-cell stimulation: an abundance of B7s. Nat. Med. 1999. 5:1345-1346.
Henry, J, Miller, MM, Pontarotti, P. Structure and evolution of the extended B7 family. Immunol. Today. 1999. 20:285-288.
Chambers, CA, Allison, JP. Costim-ulatory regulation of T cell function. Curr. Opin. Cell Biol. 1999. 11:203-210.
Wakem, P, et al. Allergens and irritants transcriptionally upregulate CD80 gene expression in human keratinocytes. J. Invest. Dermatol. 2000. 114:1085-1092.
Donepudi, M, Raychaudhuri, P, Bluestone, JA, Mokyr, MB. Mechanism of melphalan-induced B7-1 gene expression in P815 tumor cells. J. Immunol. 2001. 166:6491-6499.
Satoh, S, et al. Glomerular endothelium exhibits enhanced expression of costimulatory adhesion molecules, CD80 and CD86, by warm ischemia/reperfusion injury in rats. Lab. Invest. 2002. 82:1209-1217.
Takeda, K, Kaisho, T, Akira, S. Toll-like receptors. Annu. Rev. Immunol. 2003. 21:335-376.
Beutler, B, Rietschel, ET. Innate immune sensing and its roots: the story of endotoxin. Nat. Rev. Immunol. 2003. 3:169-176.
Mundel, P, et al. Rearrangements of the cytoskeleton and cell contacts induce process formation during differentiation of conditionally immortalized mouse podocyte cell lines. Exp. Cell Res. 1997. 236:248-258.
Jat, PS, et al. Direct derivation of conditionally immortal cell lines from an H-2Kb-tsA58 transgenic mouse. Proc. Natl. Acad. Sci. U. S. A. 1991. 88:5096-5100.
Wang, Z, et al. α3β1 integrin regulates epithelial cytoskeletal organization. J. Cell Sci. 1999. 112:2925-2935.
Kretzler, M, et al. Novel mouse embryonic renal marker gene products differentially expressed during kidney development. Am. J. Physiol. 1996. 271:F770-F777.
Mundel, P, et al. Synaptopodin: an actin-associated protein in telencephalic dendrites and renal podocytes. J. Cell Biol. 1997. 139:193-204.
Reiser, J, et al. Regulation of mouse podocyte process dynamics by protein tyrosine phosphatases. Kidney Int. 2000. 57:2035-2042.
Hamano, Y, et al. Determinants of vascular permeability in the kidney glomerulus. J. Biol. Chem. 2002. 277:31154-31162.
Borriello, F, et al. B7-1 and B7-2 have overlapping, critical roles in immunoglobulin class switching and germinal center formation. Immunity. 1997. 6:303-313.
Bosma, GC, Custer, RP, Bosma, MJ. A severe combined immunodeficiency mutation in the mouse. Nature. 1983. 301:527-530.
Barisoni, L, Kriz, W, Mundel, P, D'Agati, V. The dysregulated podocyte phenotype: a novel concept in the pathogenesis of collapsing idiopathic focal segmental glomerulosclerosis and HIV-associated nephropathy. J. Am. Soc. Nephrol. 1999. 10:51-61.
Sedwick, CE, et al. TCR, LFA-1, and CD28 play unique and complementary roles in signaling T cell cytoskeletal reorganization. J. Immunol. 1999. 162:1367-1375.
Mihara, M, et al. CTLA4Ig inhibits T cell–dependent B-cell maturation in murine systemic lupus erythematosus. J. Clin. Invest. 2000. 106:91-101.
Borriello, F, et al. Characterization of the murine B7-1 genomic locus reveals an additional exon encoding an alternative cytoplasmic domain and a chromosomal location of chromosome 16, band B5. J. Immunol. 1994. 153:5038-5048.
Whiteside, C, Prutis, K, Cameron, R, Thompson, J. Glomerular epithelial detachment, not reduced charge density, correlates with proteinuria in adriamycin and puromycin nephrosis. Lab. Invest. 1989. 61:650-660.
Cheng, ZZ, et al. Hypercholesterolemia is a prerequisite for puromycin inducible damage in mouse kidney. Kidney Int. 2003. 63:107-112.
Saleem, MA, et al. Co-localization of nephrin, podocin, and the actin cytoskeleton: evidence for a role in podocyte foot process formation. Am. J. Pathol. 2002. 161:1459-1466.
Kelley, VE, Cavallo, T. An ultrastructural study of the glomerular slit diaphragm in New Zealand black/white mice. Lab. Invest. 1976. 35:213-220.
Putaala, H, Soininen, R, Kilpelainen, P, Wartiovaara, J, Tryggvason, K. The murine nephrin gene is specifically expressed in kidney, brain and pancreas: inactivation of the gene leads to massive proteinuria and neonatal death. Hum. Mol. Genet. 2001. 10:1-8.
Schnabel, E, Anderson, JM, Farquhar, MG. The tight junction protein ZO-1 is concentrated along slit diaphragms of the glomerular epithelium. J. Cell Biol. 1990. 111:1255-1263.
Tamir, A, Eisenbraun, MD, Garcia, GG, Miller, RA. Age-dependent alterations in the assembly of signal transduction complexes at the site of T cell/APC interaction. J. Immunol. 2000. 165:1243-1251.
Wulfing, C, Davis, MM. A receptor/cytoskeletal movement triggered by costimulation during T cell activation. Science. 1998. 282:2266-2269.
Bromley, SK, et al. The immunological synapse. Annu. Rev. Immunol. 2001. 19:375-396.
Shaw, AS, Allen, PM. Kissing cousins: immunological and neurological synapses. Nat. Immunol. 2001. 2:575-576.
Grakoui, A, et al. The immunological synapse: a molecular machine controlling T cell activation. Science. 1999. 285:221-227.
Acuto, O, Cantrell, D. T cell activation and the cytoskeleton. Annu. Rev. Immunol. 2000. 18:165-184.
Hirokawa, M, Kuroki, J, Kitabayashi, A, Miura, AB. Transmembrane signaling through CD80 (B7-1) induces growth arrest and cell spreading of human B lymphocytes accompanied by protein tyrosine phosphorylation. Immunol. Lett. 1996. 50:95-98.
Grohmann, U, et al. CTLA-4-Ig regulates tryptophan catabolism in vivo. Nat. Immunol. 2002. 3:1097-1101.
Kaisho, T, Takeuchi, O, Kawai, T, Hoshino, K, Akira, S. Endotoxin-induced maturation of MyD88-deficient dendritic cells. J. Immunol. 2001. 166:5688-5694.
Doughtie, DG, Rao, KR. The syncytial nature and phagocytic activity of the branchial podocytes in the grass shrimp, Palaemonetes pugio. Tissue Cell. 1981. 13:93-104.
Pacquement, H, et al. Nephrotic syndrome and B leukemia [In French]. Arch. Fr. Pediatr. 1989. 46:741-742.
De Gaudio, AR, Adembri, C, Grechi, S, Novelli, GP. Microalbuminuria as an early index of impairment of glomerular permeability in postoperative septic patients. Intensive Care Med. 2000. 26:1364-1368.
Eddy, AA, Symons, JM. Nephrotic syndrome in childhood. Lancet. 2003. 362:629-639.
Version 1 (May 15, 2004): No description


my co-teachers are TOO nice!
What book for after school?
National Pension--How much do you get back?
Middle School Grade 1, lesson 8 - "Break a leg" Any tips/ideas? Help!
Grade 4 Look and Speak?
Help please for a 5th grade class with one student.
Solutions for uncooperative 3rd grade MS students?


Spectacular in its assertive diversity. Some acidity on the tongue, juicy fruitiness like biting into a white fleshed peach, a toasty note on the finish, an orchard of nuance. The vanilla is muted. An uninhibited and delicious blend.
Mainly Chenin Blanc, but Roussanne, Viognier and Marsanne round out the mix. I can see this working in the Okanagan as well as it does in South Africa; someone in Naramata should give it a shot. Food versatility through the roof. Not your Tuesday night house white (given the price point), but what a shame. Refreshing delectability.
Price: A rather steep $39 at New District.
Market Liquidity: Gobs of gorgeousness.


Next Article Shocker! California Dems Decline To Endorse Dianne Feinstein For Re-Election! Meaning?
RESISTANCE WATCH-In a meeting about school safety and guns, President Donald Trump pivoted to some favorite targets: California and immigration. He said he might remove immigration and border officers from the Golden State and predicted that the result would be "a crime mess like you've never seen," CNN and other news outlets reported today.
(Elizabeth Aguilera, Health and Welfare Reporter, is an award winning multi-media journalist who will cover health and social services for CALmatters, where this was first posted.) Prepped for CityWatch by Linda Abrams.


These tips are about the best way to restore scars and paint chips around the home. For small holes, like those created by a doorknob, a patch package may be used. Simply adhere the patch to the wall and cover with drywall compound. For a knock-down finish – one where a trowel presses joint compound right down to type a texture that's extra mottled than a flat wall – flippantly flatten the particles with a knife as the compound combination begins to dry.
Gently sand surface until easy with the wall. Picture 1: Drive a popped nail below the floor of the drywall with a hammer and a nail set. There are every kind of ways to patch doorknob-measurement holes. Screw in two wood boards behind the drywall, one at the prime and one at the bottom of the hole. For medium holes, use a drywall metal patch. While you peel off previous adhesive or self-sticking image hangers, you usually tear off the top layer of drywall paper, leaving fuzzy brown paper exposed.
Set up wooden cleats towards the studs on either side of the opening to assist the new drywall's vertical edges (picture 1). Use scrap wooden akin to 1×2 furring for smaller repairs; if you are replacing a big sheet of drywall, reinforce the opening with 2×3 lumber. Discard the drywall and remove all exposed drywall nails or screws. However take care not to crush the drywall core.
Cowl the outlet or dent with quick-drying spackle to convey the spackle level with the drywall floor and let it dry 24 hours – or the time really helpful by the manufacturer's directions. Use a pair of C-clamps to lock the backing board to the encompassing drywall. As for materials, drywall compound, mesh tape, paper tape, drywall nails and screws take care of most repairs.
Let dry and sand easy. Apply joint compound to each side of the nook, overlaying the bead patch to smooth tough edges and cover any seams, feathering the sides. Field, and the compound hardens rapidly, so you'll be able to apply one other coat straight away. Place a taping knife or skinny plywood behind the tool to forestall extra injury to the wall — be particularly careful not to lever towards unsupported drywall or the device will punch a hole that can require a extra intensive restore.


— the true sonic beauty of Japan.
a master traditional Japanese percussionist.
We recorded 56 essential instruments for this library. From the Kabuki and Noh hayashi flute-and-drums ensemble, we sampled the kotsuzumi (an hourglass-shaped hand drum), the otsuzumi (a large hand drum), the shimedaiko (a pitched drum), the Noh flute, and various kakegoe calls (yo and ho calls). And we selected the most necessary percussive "noisemakers" to any Kabuki performance, including the odaiko (a large drum), daibyoshi (a short-bodied drum), okedoh (a barrel drum), mamedaiko (the "imperial" drum), uchiwadaiko (a "fan" drum), atarigane (a saucer-shaped gong), chanchiki (another round bell), soban (a smaller gong), ekiro (a doughnut-shaped horse bell), music boxes, hontsurigane (a large bell), dora (a gong), and mokugyo (an ornate wooden gong). These have been paired with ashibyoshi foot beats and tsukeuchi sound-effect clappers that add drama to a powerful Kabuki performance.
We were extremely fortunate to have someone as renowned as Takinojo Mochizuki perform these instruments for our recording sessions. Without his precise mastery of these instruments and his vibrant, lively playing style, we could have never reproduced the diverse expressiveness these instruments are capable of.
All instruments were recorded with multiple microphones. Direct 1/2, Overhead (stereo), and Room (stereo) captures enable simple but effective mixing.
Takinojo specializes in playing traditional Japanese percussion instruments such as the kotsuzumi and other hand drums. Performing in a variety of styles, from Kabuki music to modern, he is an undisputable master of Japanese folk music and folk instruments.
While best known for its gentle tapping sound, the kotsuzumi is in fact capable of myriad expressions. This library captures the full range of its voices, from crisp hand taps to detailed portamento slides controlled by grasping the drum's cords.
The odaiko can express the wind, snow, water, and other natural elements as well as the inner thoughts and feelings of the actors on stage. It can even imitate the spookiness of a ghost's entrance.
Our collection features odaiko performances played with nagabachi (long, tapered sticks for sound effects) and yukibai (a special stick to imitate the sound of snow), giving music producers the scope to recreate nearly every odaiko nuance.
The library includes the tsukeuchi sound-effect clappers that heighten the actors' dramatic poses and the ashibyoshi foot beats that match dance steps.
The hall used for recording the tsukeuchi, ashibyoshi, hyoushigi, and other sounds that require the hall's ambience.
We recorded numerous Noh flute phrases, such as hishigi and ashirai, and the "iyoh" and "ho" calls (27 types and 148 variations) of the ensemble players, which are essential to the nagauta (long epic songs) that form the basis of Noh and Kabuki performances.
The collection contains many rare instruments, such as the ekiro (a doughnut-shaped horse bell), music boxes, matsumushi (a small flat gong), fusegane (an altar gong), and hontsurigane (a large bell), used for Kabuki sound effects.
The instruments were recorded with up to 99 velocity layers to capture the full range of their expressiveness.
Recording left and right hand hits separately for instruments played with both hands enables the recreation of realistic dynamics. Drum rolls also sound very natural.
The recording was done in an anechoic room similar to the bamboo-curtained orchestra room where the instruments are played in a Kabuki theater. Reverb can be added as needed because the recordings preserve the true character of the instruments' sound.
To record the true essence of the instruments' sound, we used well-kept Neuman U 47, U 67, and Korby KAT67 condenser microphones and RCA 44BX and 77DX ribbon microphones. A vintage NEVE 5315 microphone preamp translated the microphone signals to the board.
The BFD2 library contains 160 grooves performed on MIDI drums by Takinojo Mochizuki, giving you instant access to real nagauta and Noh performance rhythm patterns.
This pack is compatible with BFD2 and BFD Eco. If you have BFD2, you can take advantage of all the library's features, including full 24-bit resolution, up to 99 velocity layers, special presets, kits, and grooves. The pack is available for download from the FXpansion and Media Integration web stores. A special packaged version is planned for the Japan market as well.
See the FXpansion site for details on using BFD2.1 and BFD Eco.
Kabuki & Noh Percussion fills a long-held need of many music creators for a library of traditional Japanese instruments. This special collection contains a wide variety of Japanese instruments used in Noh and Kabuki that have yet to surface anywhere else.
Overlooked instruments include tsukeuchi (wooden clappers that are struck on a wooden plank to produce sounds that accompany Kabuki actors' dramatic poses) and ashibyoshi (floor-shaking foot stomps). Naturally, the library includes tsuzumi hand drums and other familiar Kabuki sounds. And because the library was made completely in Japan with the requisite attention to the finest of details, this is a definitive Japanese sound library.
All instruments were recorded at full 24-bit/96 kHz fidelity with an assortment of vintage microphones and an old Neve 5315 console. The result is a truly musical sound library owing to the producer's meticulous research and scrupulous attention to sound quality.
Kabuki & Noh Percussion is sure to satisfy not only musicians but also sound effect producers and others that work in the realm of sound. It will likely prove hugely useful to students of Japanese music as well.
This sound library will definitely become the world standard in Japanese instruments.
Michiaki formed the musical unit Dido with Shizuru Ohtaka in 1989 and released the 1989 album "Pajina" on Polystar and the 1994 album "Ksana" on King Records. Today, he arranges and produces songs for other artists and composes music for TV commercials and plays. He has composed more than 2,000 songs for commercials.
Recently, he has collaborated on many musical works with a number folk instrument players and traditional Japanese instrumentalists such as Chen Min, Yilana, and Hiromitsu Agatsuma. He also works as an arranger with the Hiroshima Symphony Orchestra, the Nara Philharmonic Orchestra, and the Gunma Symphony Orchestra.
Michiaki's activities span the entire gamut from sound effects to full orchestras.


text:
Mark Hummel's Blues Harmonica Blow-Out Celebrates 25 Years With Portland Concert; Feb 3rd
2016 January, PDX Blues Event
Always a must-see show whenever it comes to town, Mark Hummel's Blues Harmonica Blow-Outs are consistently one of the best blues groupings to be found. For the past twenty-five years, Hummel has brought together the top names in harmonica to be a part of these performances, and for the silver anniversary showcase he has amassed one of the best yet!
Alongside the Blues Music Award harmonica player Mark Hummel himself the featured line-up this year will be: 82-year-old swamp legend and Blues Hall of Fame member Lazy Lester, Blues Music Award Entertainer of the Year winner and Portland's own Curtis Salgado, rising blues star and one of the most innovative harp players around Jason Ricci, the Bay Area's Mumbai-born harper Aki Kumar, and San Diego's Big Jon Atkinson who is a triple threat playing not only harmonica, but guitar and drums as well. The blowers will be backed by two of the toughest, most tuneful guitarists going, Anson Funderburgh and Little Charlie Baty, while the rhythm section will be held down by California bassist RW Grigsby and legendary Texas drummer Wes Starr. Definitely a line-up for the ages and a show not to be missed!
The Blues Harmonica Blow-Out will be appearing at The Aladdin Theater in Portland on Wednesday, February 3 for an 8:00 pm show. Minors are welcome to the show when accompanied by a parent or legal guardian. Tickets are $32.50 in advance through TicketFly.com or at the Aladdin box office; $35.00 day of show at the venue.
December 29, 2015 /by Wendy Schumer
https://archive.cascadebluesassociation.org/wp-content/uploads/2015/12/Mark-Hummel-photo-by-Greg-Johnson.jpg 600 450 Wendy Schumer https://archive.cascadebluesassociation.org/wp-content/uploads/2013/03/CBA_Logo-ne3w-300x281.png Wendy Schumer2015-12-29 10:00:112020-02-15 07:49:08Mark Hummel's Blues Harmonica Blow-Out Celebrates 25 Years With Portland Concert; Feb 3rd
What's on Tap for January General Membership Meeting … Eagle Rock Gospel Singers To Perform At Mississippi Studios; Jan 31st

redpajama_set_name: RedPajamaCommonCrawl


ran with 3 missionaries 2.07 miles in 19:34.
running in same shoes while in the US. not used to the hills at all. total time 37:32. splits 9:40, 9:04, 8:26, 9:05, 1:17.
long run... 3.6 miles with Jeff. total time 72:14. not used to the hills. splits 9:06, 9:41, 9:23, 9:13, 9:34, 8:45, 8:39, 7:53.
total time 31:51. easy run 9:37, 9:03, 8:53, 4:19.
ran with President and Sister Kimbal from Pachuca and Elder Pieper and Elder Valenzuela. total time 36:02. splits 9:10, 9:09, 8:52, 8:43, 0:08.
total time 75:24 ran first 2.5 with missioneries. splits 9:24, 9:09, 9:03, 8:58, 9:10, 8:53, 8:50, 8:46, 3:09.
total time 36:20. first 2.5 with a mssionary. splits 9:52, 9:50, 9:52, 6:46. delberately slow.
total time 43:34. first 2.5 with a missionary. not feeling well put tried to push. splits 9:14, 9:27, 9:12, 8:55, 6:47.
easy run. total time 31:52. splits 8:56 (too fast), 9:12, 9:21, 4:22.


Shawn is the project lead for SiteFarm and supervises the IET Professional Services - Web Development team.
First tier support for SiteFarm.
On-boarding, service, support, technical documentation, development, troubleshooting, and training.


text:
\section{Introduction}\label{sec:introduction}

The current dominant paradigm for machine learning is to run an algorithm on a given dataset to produce a trained model specifically for a particular purpose; this is \emph{isolated learning}~\citep[p.~150]{lll_book}.
In isolated learning, the model is unable to retain and accumulate the knowledge it has learned before.
When a stream of tasks are joined to be trained sequentially, isolated learning faces \emph{catastrophic forgetting}~\citep{mccloskey1989catastrophic} due to a non-stationary data distribution that biases the model (left figure of Figure~\ref{fig:framework}).
In contrast, lifelong learning is designed to address a stream of tasks by accumulating interconnected knowledge between learned tasks and retaining the performance of those tasks.
A human easily achieves lifelong learning, but this is nontrivial for a machine; thus lifelong learning is a vital step toward artificial general intelligence.

In this paper, we focus on lifelong language learning, where a machine achieves lifelong learning on a stream of natural language processing (NLP) tasks.
To the best of our knowledge, lifelong language learning has been studied in only a few instances;
for sentiment analysis~\citep{chen-etal-2015-lifelong,8101496}, conversational agents~\citep{LLL_chatbot}, word representation learning~\citep{LLL_word}, sentence representation learning~\citep{Liu2019ContinualLF}, text classification, and question answering~\citep{d2019episodic}.
However, in all previous work, the tasks in the stream are essentially the same task but in different domains.
To achieve lifelong language learning on fundamentally different tasks, we propose LAMOL~--- LAnguage MOdeling for Lifelong language learning.

\input{figures/framework}

It has been shown that many NLP tasks can be considered question answering (QA)~\citep{McCann2018decaNLP}.
Therefore, we address multiple NLP tasks with a single model by training a language model (LM) that generates an answer based on the context and the question.
Treating QA as language modeling is beneficial because the LM can be pre-trained on a large number of sentences without any labeling~\citep{radford2019language}; however, this does not directly solve the problem of LLL.
If we train an LM on a stream of tasks, catastrophic forgetting still occurs.
However, as an LM is intrinsically a text generator, we can use it to answer questions while generating pseudo-samples of the previous task to be replayed later.
LAMOL is inspired by the data-based approach for LLL in which a generator learns to generate samples in previous tasks (middle of Figure~\ref{fig:framework})~\citep{shin2017continual,kemker2017fearnet}.
In contrast to previous approaches, LAMOL needs no extra generator (right of Figure~\ref{fig:framework}).
LAMOL is also similar to multitask training, but the model itself generates data from previous tasks instead of using real data.

Our main contributions in this paper are:
\begin{itemize}[leftmargin=*]
    \item We present LAMOL, a simple yet effective method for LLL. Our method has the advantages of no requirements in terms of extra memory or model capacity. We also do not need to know how many tasks to train in advance and can always train on additional tasks when needed.
    \item Experimental results show that our methods outperform baselines and other state-of-the-art methods by a considerable margin and approaches the multitasking upper bound within 2--3\%.
    \item Furthermore, we propose adding task-specific tokens during pseudo-sample generation to evenly split the generated samples among all previous tasks. This extension stabilizes LLL and is particularly useful when training on a large number of tasks.
    \item We analyze how different amounts of pseudo-samples affect the final performance of LAMOL, considering results both with and without the task-specific tokens.
    \item We open-source our code to facilitate further LLL research.
\end{itemize}

\section{Related Work}
Lifelong learning research is based on regularization, architecture, or data.
Here is a brief survey of works in these three categories.

\subsection{Regularization-based methods}

In this approach, a constraint, i.e., a regularization term, is added to minimize deviation from trained weights while updating the weights in a new task.
Most regularization based methods estimate the importance of each parameter and add the importance as a constraint to the loss function.
Elastic weight consolidation (EWC)~\citep{kirkpatrick2017overcoming} calculates a Fisher information matrix to estimate the sensitivity of parameters as importance.
Online EWC~\citep{schwarz2018progress} is a transformed version of EWC.
Instead of tracking the importance of parameters for each task, online EWC simply accumulates the importance of the stream of tasks.
Synaptic intelligence (SI)~\citep{zenke2017continual} assigns importance to each parameter according to its contribution to the change in the total loss.
Memory aware synapses (MAS)~\citep{aljundi2018memory} estimate importance via the gradients of the model outputs.
In contrast to estimating the importance of weights, incremental moment matching (IMM)~\citep{lee2017overcoming} matches the moment of weights between different tasks.

\subsection{Architecture-based methods}

For this category, the main idea is to assign a dedicated capacity inside a model for each task.
After completing a task, the weights are frozen and may not be changed thereafter.
Some methods allow models to expand, whereas some fix the size but must allocate capacity for tasks at the beginning.
Progressive neural networks~\citep{rusu2016progressive} utilize one column of the neural network per task.
Once a new task is trained, progressive neural networks augment a new column of the neural network for the task while freezing the past trained columns.
Columns that have been frozen are not allowed to change but are connected to the new column to transfer knowledge from old tasks.
Towards Training Recurrent Neural Networks for Lifelong Learning~\citep{sodhani2018training} unifies Gradient episodic memory~\citep{lopez2017gradient} and Net2Net ~\citep{chen2015net2net}.
Using the curriculum-based setting, the model learns the tasks in easy-to-hard order.
The model alleviates the forgetting problem by GEM method, and if it fails to learn the current task and has not been expanded yet, the model will expand to a larger model by the Net2Net approach.

PathNet~\citep{fernando2017pathnet} reuses subsets of a neural network to transfer knowledge between tasks. Unlike progressive neural networks, PathNet does not allow the model to expand. Instead, it builds a huge fixed-size model composed of a neural network and paths between different layers of the neural networks. While training a task, it selects the best combination of neural networks and paths for that particular task.
Similar to progressive neural networks, selected parts are fixed to allow only inference and not training.
Inspired by network pruning, PackNet~\citep{mallya2018packnet}  prunes and re-trains the network iteratively to pack numerous tasks into a single huge model.

This category has some drawbacks.
When resources are limited, model expansion is prohibited.
Also, some architecture-based methods require the number of tasks in advance to allocate the capacity for the tasks, which greatly reduces their practicality.

\subsection{Data-based methods}

This method restricts weights through the data distribution of old tasks. One data-based approach keeps a small amount of real samples from old tasks, and the other distills the knowledge from old data and imagines pseudo-data of old tasks later on. While training a new task, the data or pseudo-data is used to prevent weights from greatly deviating from the previous status.

Gradient episodic memory (GEM)~\citep{lopez2017gradient} preserves a subset of real samples from previous tasks.
Utilizing these real samples during optimization helps somewhat to constrain parameter gradients.
Averaged-GEM (A-GEM)~\citep{chaudhry2018efficient} is a more efficient version of GEM which achieves the same or even better performance than the original GEM. Learning without forgetting~\citep{li2017learning} minimizes the alteration of shared parameters by recording the outputs from old task modules on data from the new task before updating. \citet{shin2017continual} and \citet{kemker2017fearnet} encode data from old tasks into a generative model system. The latter imitates the dual-memory system of the human brain, in that the model automatically decides which memory should be consolidated.
Both methods replay pseudo-data of previous tasks using the generative model during training.

\citet{d2019episodic} investigates the performance of the episodic memory system on NLP problems.
It distills the knowledge of previous tasks into episodic memory and replays it afterward.
This work evaluates the method on two streams of tasks: question answering and text classification.\\

\section{LAMOL}\label{sec:our}
A pre-trained LM can generate a coherent sequence of text given a context.
Thus, we propose LAMOL, a method of training a single LM that learns not only to answer the question given the context but also to generate the context, the question, and the answer given a generation token.
That is, in LAMOL, a model plays the role of both LM and QA model.
Hence, answering questions and generating pseudo-old samples can both be done by a single model.
During LLL, these pseudo-old samples are trained with new samples from new tasks to help mitigate catastrophic forgetting.
\input{figures/training}

\subsection{Data formatting} \label{subsec:data_format}

Inspired by the protocol used by decaNLP~\citep{McCann2018decaNLP}, samples from the datasets we used are framed into a SQuAD-like scheme, which consists of context, question, and answer.
Although the LM is simultaneously a QA model,
the data format depends on the training objective.
When training as a QA model, the LM learns to decode the answer after reading the context and question.
On the other hand, when training as an LM, the LM learns to decode all three parts given a generation token.

In addition to context, question, and answer, we add three special tokens:
\begin{description}
    \item[ANS] Inserted between question and answer. As the context and question are known during inference, decoding starts after inputting ANS.
    \item[EOS] The last token of every example. Decoding stops when EOS is encountered.
    \item[GEN] The first token during pseudo-sample generation. Decoding starts after inputting GEN.
\end{description}
The data formats for QA and LM training are shown in Figure~\ref{fig:training}.

\subsection{Training}
Assume a stream of tasks $\{T_1, T_2, \dots\}$, where the number of tasks may be unknown.
Directly training the LM on these tasks sequentially results in catastrophic forgetting.
Thus, before beginning training on a new task $T_i, i > 1$, the model first generates pseudo samples $T^{'}_i$
by top-$k$ sampling that represent the data distribution
of previous tasks $T_1, \dots, T_{i-1}$.
Then, the LM trains on the mixture of $T_i$ and $T^{'}_i$.
To balance the ratio between $|T_i|$ and $|T^{'}_i|$, the LM generates $\gamma |T_i|$ pseudo samples, where $|T_i|$ denotes the number of samples in task $T_i$ and $\gamma$ is the sampling ratio.
If the generated sample does not have exactly one ANS in it, then the sample is discarded.
This happens in only 0.5\%-1\% of generated samples.

During training, each sample is formatted into both the QA format and the LM format.
Then, in the same optimization step, both formats are fed into the LM to minimize the QA loss $L_{\text{QA}}$ and LM loss $L_{\text{LM}}$ together. Overall, the loss is $L = L_{\text{QA}} + \lambda L_{\text{LM}}$, where $\lambda$ is the weight of the LM loss.

\subsection{Task-specific tokens}
Using the same GEN token for all tasks is problematic when training for many tasks because the portion of old tasks decreases exponentially in theory.
For instance, if $\gamma = 0.01$, then the portion of the first task when training the second task is about 1\%, but is only about 0.01\% when training the third task.
This issue is definitely harmful to LLL.
To mitigate this, we can choose to replace the GEN token with a task-specific token for each task to inform the model to generate pseudo-samples belonging to the specific task.
Under this setup, all previous tasks have the same share of the $\gamma |T_i|$ generated pseudo samples.
That is, when beginning training for the $i$-th task $T_i$, we generate $\frac{\gamma}{i-1} |T_i|$ for the previous $i-1$ tasks.
Note that as each task uses a specific token, the vocabulary size and the embedding weight of the LM increase slightly as more tasks are trained.

\begin{comment}
\subsubsection{Definition}
We annotate our method as \textbf{$LAML_{Task|Gen}^N$}.
$N$ means the \% of current task size we generate.
$Task|Gen$ corresponds to what kind of generation token we use. $Task$ is equivalent to task-specific token while $Gen$ means the constant generation token .
For all tasks, We simply fit the model on training dataset without any supervision by development set.
\end{comment}

\section{Experiment Setup} \label{sec:expsetup}

\subsection{Tasks, datasets, and metrics}

We collect five disparate tasks mentioned in decaNLP~\citep{McCann2018decaNLP}: question answering, semantic parsing, sentiment analysis, semantic role labeling, and goal-oriented dialogue, with a dataset for each task.

Furthermore, to compare our method with~\citet{d2019episodic}, we conducted experiments on four text classification tasks: news classification, sentiment analysis, Wikipedia article classification, and question-and-answer categorization with five datasets. We use the procedure from~\citet{d2019episodic} to produce equal-sized datasets.

We do not train on all datasets from both papers due to a lack of computational resources.
For each task, there is a corresponding evaluation metric.
Table~\ref{tab:data_attrs} contains a summary of tasks, datasets, and metrics.
Additional details are provided in Appendix~\ref{appendix:dataset}.
Note that the score of any metric lies between 0 and 100\%.

\input{tables/data_attrs}
\input{tables/single_task}

\subsection{Methods to be compared}

All methods use the smallest pre-trained GPT-2 model~\citep{radford2019language}\footnote{https://github.com/huggingface/pytorch-transformers} as the LM.
Each task is trained for nine epochs; greedy decoding is applied during inference.

\begin{itemize}[leftmargin=*]

    \item \textbf{LAMOL} In all experiments, $k = 20$ in top-$k$ sampling and $\lambda = 0.25$ for weight of the LM loss are set.
$\text{LAMOL}^{\gamma}_{\text{GEN}}$ denotes LAMOL with a sampling ratio of $\gamma$, and the same GEN token is used for all tasks.
If the task-specific tokens are used, GEN is replaced by TASK.

    \item \textbf{Keep real data} Pseudo-samples are replaced by real samples from previous tasks.
The quantity of real samples is equally split between previous tasks.
This approach can be considered the upper bound of LAMOL.
We denote it as $\text{LAMOL}^{\gamma}_{\text{REAL}}$.

    \item \textbf{Fine-tune}
The model is directly fine-tuned on the stream of tasks, one after another.

    \item \textbf{Multitask learning} All tasks are trained simultaneously.
Multitask learning is often seen as an upper bound of lifelong learning.
In addition, it is also used to determine whether forgetting is caused by a lack of model capacity.

    \item \textbf{Regularization-based methods}
Online EWC~\citep{schwarz2018progress} and MAS~\citep{aljundi2018memory} are compared.
They are chosen because they are more computationally efficient than SI~\citep{zenke2017continual} and more memory efficient than
IMM~\citep{lee2017overcoming}.
Additionally, experiments such as~\cite{elhoseiny2018exploring} show that MAS has better performance overall.

    \item \textbf{Gradient Episodic Memory (GEM)} When training each task, we randomly sample data from previous task with the amount equivalent to 5\% of the current task size into the memory. In each optimization step, the GEM~\citep{lopez2017gradient} approach retrieves all the data in the memory to calculate the gradients for the previous tasks.

    \item \textbf{Improved memory-based parameter adaptation (MBPA++)} Sparse experience replay and local adaptation for LLL as proposed in~\citet{d2019episodic}. We also re-implement the paper and report better scores using different hyperparameters.

\end{itemize}

\section{Experimental Results}

\subsection{Single Task}
To establish a reference on the capability of the GPT-2 model on every dataset, we trained the model on each dataset independently.
The results are shown in Table~\ref{tab:single_task}.
We observe that the performance of the GPT-2 model is actually quite good, even beating the BERT-based model~\citep{d2019episodic} on text classification datasets by a large margin.
Thus, the GPT-2 model has the potential for superior LLL performance, as long as we can prevent catastrophic forgetting.

\input{tables/small_perm}
\input{tables/five_tasks}

\subsection{SST, QA-SRL, and WOZ Tasks}
For an initial understanding of the performance on all of the methods and the effect of task order, we first conducted a small-scale experiment on three small datasets: SST, QA-SRL, and WOZ.
We trained all but the the multitasked method on all six permutations of the task order.
The final score for each order was obtained by evaluating the model at the conclusion of the training process.
The results are shown in Table~\ref{tab:small_perm}; we make several observations.
Note that LAMOL with $\gamma = 0$ is not the same as Fine-tuned, as the LM loss is still optimized.
\begin{itemize}[leftmargin=*]
    \item Fine-tuned, EWC, MAS, and LAMOL with $\gamma = 0$ show similar performance and are much worse than LAMOL with $\gamma > 0$.
    \item $\text{LAMOL}^{0.2}_{\text{GEN}}$, our best performing method, is only 1.8 percent away from Multitasked, which implies almost no forgetting during LLL.
    \item The order of the tasks is crucial to the performance.
    For instance, the WOZ score drops significantly after training other tasks.
    Thus, if WOZ is not the last task, the performance is usually noticeably worse.
    \item When using LAMOL, the performance of old tasks maintains almost the same level throughout the training process.
    When the sampling ratio $\gamma$ is increased, the performance also increases, especially when increased from 0 to 0.05.
    \item When $\gamma = 0$, adding task-specific tokens harms performance, because the model must fit additional special tokens that are useless.
    Adding task-specific tokens is also not helpful if $\gamma = 0.2$. We believe that 0.2 is enough for three tasks; thus task-specific tokens are redundant.
    However, when $\gamma = 0.05$, task-specific tokens are beneficial because the tokens are needed to help retain a substantial presence of the first task when training the third task.
    \item  We see that a better LLL method usually has a smaller standard deviation, which implies that it is effected less by task order. Adding task-specific tokens also has a stabilizing effect.
\end{itemize}

The complete forgetting progress is illustrated in Appendix~\ref{appendix:small_perm}.
Clearly, Fine-tuned, EWC, MAS, $\text{LAMOL}_{\text{GEN}}^0$, and $\text{LAMOL}_{\text{TASK}}^0$ reveal similar patterns.
However, the proposed LAMOL with $\gamma > 0$ displays the ability to retain its learned knowledge.
In the case of
WOZ $\rightarrow$ SRL $\rightarrow$ SST,
the WOZ score even increases after training the third task using LAMOL with $\gamma = 0.2$.

\input{figures/five_tasks_cmp_gen_size}

\subsection{Five decaNLP Tasks}
Here, we train the following five tasks sequentially: SQuAD, WikiSQL, SST, QA-SRL, and WOZ.
Given the limited computing resources, we explore only one task order: from large to small tasks, according to the number of training samples.

As shown in Table~\ref{tab:five_tasks}, LAMOL outperforms all baselines by a large margin and on average approaches within 2--3\% of the multitasked upper bound.
Also, as expected, the performance of LAMOL improves as the sampling ratio $\gamma$ increases and task-specific tokens are used.

There is also a gap between our method and the method of keeping real samples.
As shown in the table, using real samples is much more sample-efficient, as 5\% of real samples beats 20\% of pseudo-samples.
This may be due to the less-than-ideal quality of the pseudo-data.
The longer the paragraphs are, the harder it is for the model to create high-quality samples.
After observing the samples generated when using task-specific tokens, we discover some ``chaos''.
That is, some examples generated by the model do not exactly correspond to the task-specific token.
This implies that the task-specific tokens are sometimes too weak to constrain the model; thus their influence is overshadowed by other tokens.
We believe that solving this problem will bring the performance when using task-specific tokens closer to using real samples; however, we leave this as future work.

Figure~\ref{fig:five_tasks} illustrates the test scores of each method on each task throughout the training.
We clearly see that when using LAMOL, the model remembers nearly perfectly.

We make several observations:
\begin{itemize}
    \item When training SQuAD, QA-SRL has not been trained yet, but the score of QA-SRL is already around 40.
    Also, when training QA-SRL, the SQuAD score revives if the model has forgotten SQuAD.
    These two facts imply that SQuAD and SRL are similar tasks, such that the model is capable of transferring knowledge from one to the other.
    \item If forward transfer exists, replaying pseudo-data also retains the forward transfer. That is, the QA-SRL score does not drop after training on WikiSQL and SST when LAMOL is used but drops significantly for other methods.
    \item The transferability between SQuAD and QA-SRL is expected.
On the other hand, the transferability between WikiSQL and QA-SRL is quite surprising; the WikiSQL score improves considerably when training on QA-SRL for Fine-tuned and MAS after WikiSQL is forgotten during SST training.
\end{itemize}

\subsection{Text classification tasks}
We compared the proposed method against the state-of-the-art MBPA++ proposed in~\citet{d2019episodic}, both by citing their original numbers and also by reproducing their methods.
We chose text classification as opposed to QA because we believe that LM has more of a disadvantage in text classification than in QA.
We compared with $\text{LAMOL}_{\text{TASK}}^{0.2}$ due to its good performance and stability.
Following their paper and testing our model on the same four kinds of task orders, the results are shown in Table~\ref{tab:text_cls}.

Our implementation results in much higher scores than the original ones.
However, the proposed $\text{LAMOL}_{\text{TASK}}^{0.2}$ still outperforms our implementation of MBPA++.

\subsection{Influence of sampling ratio $\gamma$}
As the value of $\gamma$ determines the performance of LLL,
we conducted a medium-scale experiment to understand the influence of $\gamma$ with and without task-specific tokens.
In this experiment we used WikiSQL (blue color), SST (orange), QA-SRL (green), and WOZ (red), in that training order.
The results are shown in Figure~\ref{fig:cmp_gen_size}.

Unsurprisingly, the less generation done by the model, the more likely the vanishing distribution in Section~\ref{sec:our} occurs: the model forgets how to generate previous tasks, as the ratio of previous tasks in the total dataset decreases exponentially over time.
Models using task-specific tokens mitigate this somewhat, as demonstrated in the first subgraph where the performance of $\text{LAMOL}_{\text{TASK}}^{0.03}$ is much better than that of $\text{LAMOL}_\text{GEN}^{0.03}$.

In addition, the more samples the model generates, the better the overall performance of the model.
However, this performance gain disappears when the sampling ratio $\gamma$ is around 0.1 to 0.3.

\input{tables/text_cls}

\section{Conclusion}

We propose LAMOL, a simple yet effective method for LLL based on language modeling.
A single LM achieves LLL without additional model components and without keeping old examples.
Moreover, any pre-trained LM can be used to leverage a large amount of unlabeled text to improve LLL.
Finally, more tasks can be added whenever needed.

\section*{Acknowledgement}
This work was supported by the Ministry of Science and Technology of
Taiwan.

redpajama_set_name: RedPajamaArXiv


text:
Q: Google cloud share data between VM's I wish to create a Virtual Machine on Google Compute Engine with GPU's in order to carry out certain tasks.
Now the problem is that the data to be uploaded to the VM's from my local network is huge, and I am severely restricted by my slow and unreliable internet. I am aware of file transfer between unix machines using SCP, but even that would take hours to take place, and that means my GPU's online (the most expensive components) would be sitting idle, and I would be charged for nothing.
I have thought of transferring files to the VM first and then adding GPU's, but I have been unable to edit the VM to add GPU's in such a case.
Hence, I need help with two possible solutions.
Is it possible for me to add a persistent disk to one VM, transfer data, spawn a new VM and shift the disk to the new VM? If yes then how?
OR
Is it possible to edit an existing VM to add GPU instances post it's creation? If yes then how?
Any sort of help will be appreciated. Thank You!

A: These are some suggestions:

*

*Upload your data to a Google Cloud Storage bucket and then pull data from there to the VM when needed (this will be relatively fast compared to copying data from your local machine into the VM). GCS also provides a FUSE tool to mount the GCS buckets on your VM and then be able to read/write data as needed. GCS buckets is the most flexible option in this list.

*You can create a persistent disk, upload the data into it once. Then attach it as needed to your GPU based VMs when needed. You can have multiple VMs attaching the same disk simultaneously in read-only mode.

*You can snapshot persistent disks, and restore a snapshot on to a new persistent disk if needed. This is more useful for backups than your primary use case.

*Consider chunking and/or sharding your data so that you can pipeline pulling data from a remote server (like GCS) when you're running your workloads on GPU in parallel.

A: If it's a batch workload and take less than 24 hours to complete, as it sounds, you should use preemptible machines to reduce the cost drastically.
The technical details are here :

*

*https://cloud.google.com/compute/docs/instances/create-start-preemptible-instance
Maybe few highcpu with the latest Intel Skylake CPU generation could work for you ? I don't think that you can preempt GPU machines.
My 2 cents.

redpajama_set_name: RedPajamaStackExchange


text:
Home News Regions Northern Virginia Sanctuary for wounded warriors to open this fall
Sanctuary for wounded warriors to open this fall
Published June 28, 2013 by Veronica Garabelli
After spending more than a year at Walter Reed Army Medical Center in Washington, D.C., Adam Popp longed for the simple things in life, like relaxing at a friend's house and watching TV. "Getting out of the hospital, that was one of my favorite things to do," says Popp, who lost his right leg while fighting in Afghanistan in 2007.
Ken and Julia Falke aim to help wounded veterans like Popp with the creation of the 37-acre Boulder Crest Retreat in the Bluemont area of Loudoun County. The free retreat, situated in the foothills of the Blue Ridge Mountains, is set to open in September. It is designed to be a place where wounded soldiers and their families can rest, play (activities range from fly fishing to archery) and plan for the future.
This isn't the Falkes' first involvement with nonprofits. They founded the EOD Warrior Foundation in 2007. It provides financial assistance to the families of EOD (Explosive Ordnance Disposal) technicians who have been injured in the line of duty. Ken Falke, himself a former bomb disposal technician, founded A-T Solutions, a counterterrorism firm in Vienna where he's currently a shareholder. EOD Warrior Foundation provided a grant that paid travel and hotel costs, so that Popp's family could visit him while he was at the hospital.
Shoulder 2 Shoulder Inc., a business Falke founded with Chris Ferguson in 2011, also helps wounded soldiers. The small startup has built Impact (Injured Military Personnel Assisting Combat Troops), a free, video-driven social network where wounded warriors share their stories to help reduce injuries on the battlefield and help other injured soldiers through the healing process.
The Falkes have been busy raising money for Boulder Crest, reaching $5 million of their $10 million capital campaign goal in 18 months. Half of the money will be used for construction, with the other half being used for operating expenses for the first six years.
Virginia companies that have donated to the project include Richmond-based Altria Group, McLean-based Capital One Financial, Bridgewater-based Dynamic Aviation and Vienna-based MicroTech.
Looking to the future, the Falkes plan to operate Boulder Crest using grants, a sustainable giving program and the proceeds from renting the retreat facilities for events.
"Any fundraising in today's economy is challenging. We've done really well," Ken Falke says of Boulder Crest's progress so far.
Calkain Cos. brokers $8.3 million, multitenant retail deal in Loudoun
MGM National Harbor unveils designs for suites at new casino resort
Graham Holdings Co. closes on sale of two Robinson Terminal sites in Alexandria


Your greatest investment should be the one you call home.
The Hamptons are one the most unique and competitive real estate markets in the world, and I understand that making a decision to buy a home here can be particularly overwhelming. Whether you are a seasoned renovator or a first time homeowner, things don't always go as planned and emotions can run high. Being well prepared and clear on your plan of action is critical for sanity's sake.
MOTIVATION: Full time residence? Summer home? Weekends Year Round? Flip or Speculative Project? Investment property only?
INVESTMENT: Long term hold? Or shorter term profit gain? The difference between these in your search is significant.
KNOW YOUR BUDGET: Be clear on what you can and want to spend. Time is money and you'll be only wasting your own if you aren't realistic about this from the very beginning.
EXPECTATIONS: What is your ideal timeline to close?
FINACING: Purchase with cash or using a lender?
A CASH DEAL: Will certainly help the deal move along more quickly and your offer will hold more weight. Be prepared to verify assets before going to contract.
MORTGAGE CONTINGENT: Being pre-approved is essential, Once your offer is accepted your lender will order an appraisal, you'll need your survey before the seller can update or obtain Certificate of Occupancy. Loan commitments generally take 45-60 days.
CLOSING COSTS: Your expenses outside of the purchase will include attorney, local taxes, new or updated survey, title insurance, property insurance. This can affect what your working budget to purchase will be.
CALL ME at 631-871-6104. I'll guide your search and do all the heavy lifting.
RESEARCH your ideal location: Be flexible! Have 1st, 2nd and 3rd favorites.
LIFESTYLE: This will decide a lot about where you should be focusing your search. Near Ocean beaches? Bay beach activities? Boating, Kayaking, Paddle boarding? Walking distance to a village, markets, restaurants? Tennis Player? Horse Country?
MARKET FORCES: Investigate Buyer's Market vs. Seller's Market. Inventory Surplus.
ONLINE LISTINGS: Trust your agent who already knows the properties and the nuances of the neighborhoods but do your own search too. You know what you like but be open to suggestions - the listing photos don't always tell the full story.
SCHEDULE SHOWINGS: Are you local? You can benefit from weekday showings. Plan several visits if you are out of the area. Locating your dream property can take time and is best when not done in a stressful timeframe.
OPEN HOUSES: Go even if its above your budget, you can learn a lot about what an area offers, and which home components are 'must-haves' and which are just 'wish-list' items. Be sure to let the Listing Broker know you have an agent already!
IDENTIFY: When you think you have found that right property – act decisively.
VALUE: Analyze the Value vs. Listing Price with your agent. Review comps and what external expenses such as renovation/landscaping/ new pool will cost.
OFFER: Strategize your opening bid carefully. In this highly competitive market being foolish can cost you your ideal property. Your agent will present the offer along with all terms of the bid including, any contingencies, financing, and closing timeline.
NEGOTIATION: This is where things can get stressful. Keep you head and consider both sides of the deal by trying to imagine the other position. Unreasonable demands are rarely welcome. Keep the dialog open with your agent.
ACCEPTED OFFER: Hire an attorney. Best to work with a locally based attorney who specializes in Hamptons property sales. Your attorney's job is vital to protecting your interests; therefore you want one who is familiar with local land use and zoning regulations. Schedule an inspection – either a home inspector or a structural engineer to provide you with details about your home and water quality and termite tests.
MEMO OF SALE: Your attorney will receive a Memorandum of Sale with terms of the deal detailed.
CONTRACT: Your attorney will represent your needs and go over the terms of the contract with you. Any concerns from the inspector's report will be addressed and outlined if necessary in the contract for the Seller to remedy before closing. An 'as-is' deal means just that- you have agreed to accept the property in the condition you found it in.
SIGNING CONTRACT: When the contract negotiation is complete the Buyer signs and delivers a contract deposit to be held in escrow by Seller's attorney. Seller's attorney now returns to your attorney the fully executed contract.
MORTGAGE: With a fully executed contract now you start securing a loan commitment. Your lender will schedule to appraise the property.
SURVEY: Typically your attorney orders a new or revised survey. Please allow 2-5 weeks for this process. The land surveyor will provide you with an accurate depiction of your properties boundaries including all structures, hardscaping and clearing calculations of your property. The new survey is certified to the new owner so this cost is the Buyer's responsibility.
C of O: In most cases, especially if a lender is involved, the Town or Village where the property is located will issue a new or updated Certificate of Occupancy. A building inspector will need access to the home to certify that all structures comply with local codes. Usually this is the responsibility of the Seller.
TITLE INSURANCE: A clean and clear title insures that you won't have any unclaimed ownerships of your new home. A one-time fee at closing covers you and your heirs as long as you own the property. The title company will conduct a comprehensive search and it's representative is present at closing to prove documentation that they have not uncovered any unwanted surprises.
UTILITIES TRANSFER: You will need the necessary utilities transferred to accounts in your name before closing. Connect with your agent and attorney to get a list of service providers These will include Electric (PSEGLI), Fuel (independent oil providers or National Grid for Natural Gas), Suffolk County Town Water, Cable/Internet (Optimum) and various other house and home caretakers. Account balances will be reconciled at closing between buyer and seller.
WALK-THROUGH: Arrange with your agent to visit the property within 24 hours or on the day of the closing to verify that any contractual contingencies have been remedied, that the overall condition is as when negotiated, and that the seller has vacated and left the home "broom clean".
CLOSING DAY: Wire transfer of purchase and closing costs. If a lender is involved there will be a lot of paperwork to sign. The seller pays the listing and selling brokers' commissions. At the end you will get a set of keys and if you are not tearing the house down you are free to move in and celebrate!