Your query has returned 105 records. HEY! Click on one of the fields in the header! The table now sorts! Click again and it sorts the other way.
Plasma Blast 1 Grass 50 Basic Rx2 1 Common [G] Sweet Scent - Heal 20 damage from 1 of your Pokemon.
Plasma Blast 3 Grass 80 Restored Rx2 W-20 2 Uncommon Prehistoric Call - Once during your turn (before your attack), if this Pokemon is in your discard pile, you may put this Pokemon on the bottom of your deck. (Ability) [1G] Spiral Drain (20) Heal 10 damage from this Pokemon.
Plasma Blast 5 Grass 100 Basic Lx2 F-20 1 Uncommon [G] Return (10) Draw cards until you have 6 cards in your hand. [1G] Energy Press (20+) Does 20 more damage for each Energy attached to the Defending Pokemon.
Plasma Blast 8 Grass 80 Stage 1 Rx2 0 Rare  Retribution (20) If an Escavalier you had in play was Knocked Out by damage from an opponent's attack during his or her last turn, put all Energy attached to the Defending Pokemon into your opponent's hand. [1G] Signal Beam (30) The Defending Pokemon is now confused.
Plasma Blast 9 Grass 170 Basic Rx2 W-20 1 Ultra-Rare Verdant Wind - Each of your Pokemon that has any G Energy attached to it can't be affected by any Special Conditions. (Remove any Special Conditions affecting those Pokemon.) (Ability) [1G] Emerald Slash (50) You may search your deck for 2 G Energy cards and attach them to 1 of your Benched Pokemon. Shuffle your deck afterward.
Plasma Blast 10 Grass 100 Basic Rx2 1 Rare  Call for Family - Search your deck for 2 G Basic Pokemon and put them onto your Bench. Shuffle your deck afterward. [1GG] Jet Impact (80+) Flip a coin. If heads, this attack does 20 more damage.
Plasma Blast 12 Fire 70 Basic Wx2 3 Common [R] Flare (10) [1R] Ember (30) Discard an Energy attached to this Pokemon.
Plasma Blast 13 Fire 100 Stage 1 Wx2 1 Rare  Solar Transporter - Reveal the top 5 cards of your deck and put all Team Plasma cards you find there into your hand. Discard the other cards. [1RR] Leech Life (50) Heal from this Pokemon the same amount of damage you did to the Defending Pokemon.
Plasma Blast 16 Water 140 Stage 2 Gx2 4 Rare Deluge - As often as you like during your turn (before your attack), you may attach a W Energy card from your hand to 1 of your Pokemon. (Ability)  Hydro Pump (60+) Does 10 more damage for each W Energy attached to this Pokemon.
Plasma Blast 18 Water 60 Basic Lx2 1 Common [WW] Bubble Beam (20) Flip a coin. If heads, the Defending Pokemon is now Paralyzed.
Plasma Blast 19 Water 90 Stage 1 Lx2 1 Uncommon [W] Sharpshooting - This attack does 30 damage to 1 of your opponent's Pokemon. (Don't apply Weakness and Resistance for Benched Pokemon.) [WW] Bubble Beam (40) Flip a coin. If heads, the Defending Pokemon is now Paralyzed.
Plasma Blast 22 Water 100 Stage 1 Mx2 2 Uncommon [1W] Powder Snow (30) The Defending Pokemon is now Asleep. [2W] Reflect Energy (60) Move a W Energy from this Pokemon to 1 of your Benched Pokemon.
Plasma Blast 26 Water 120 Stage 1 Mx2 2 Uncommon [GG] Razor Leaf (40) [1GG] Bang Heads (80) Both this Pokemon and the Defending Pokemon are now Confused.
Plasma Blast 30 Water 180 Basic Mx2 3 Ultra-Rare  Outrage (30+) Does 10 more damage for each damage counter on this Pokemon. [2WW] Giga Frost (150) Discard 2 W Energy attached to this Pokemon.
Plasma Blast 35 Psychic 100 Stage 1 Dx2 0 Rare Drifting Balloon - This Pokemon's attacks cost 1 less for each of your opponent's Team Plasma Pokemon in play. (Ability)  Derail (70) Discard a Special Energy attached to the Defending Pokemon.
Plasma Blast 36 Psychic 60 Basic Px2 1 Rare [P] Psypower - Put 3 damage counters on your opponent's Pokemon in any way you like.
Plasma Blast 38 Psychic 60 Basic Px2 1 Rare  Trading Places - Switch this Pokemon with 1 of your Benched Pokemon. [2P] Psyjamming - Move as many Special Energy attached to your opponent's Pokemon to your opponent's other Pokemon in any way you like.
Plasma Blast 46 Psychic 130 Stage 1 Dx2 4 Rare  Iron Fist of Justice (60) If you have any Team Plasma Pokemon in play, this attack does nothing. [2PP] Shadow Punch (80) This attack's damage isn't affected by Resistance.
Plasma Blast 49 Fighting 150 Stage 2 Px2 3 Rare Badge of Discipline - The damage of each of your F Pokemon's attacks isn't affected by Resistance. (Ability) [3F] Close Combat (120) Flip a coin. If tails, during your opponent's next turn, any damage done to this Pokemon by attacks is increased by 30 (after applying Weakness and Resistance).
Plasma Blast 50 Fighting 140 Stage 2 Px2 3 Rare [2F] Knock Off (60) Discard a random card from your opponent's hand. [2FF] Reinforced Lariat (80+) If this Pokemon has a Pokemon Tool attached to it, this attack does 40 more damage.
Plasma Blast 51 Fighting 100 Basic Px2 3 Common [1F] Freestyle Strike (30x) Flip 2 coins. This attack does 30 damage times the number of heads. [2F] Shoulder Throw (80-) Does 80 damage minus 20 damage for each C in the Defending Pokemon's Retreat Cost.
Plasma Blast 54 Fighting 120 Stage 1 Gx2 0 Rare [F] Acrobatics (20+) Flip 2 coins. This attack does 20 more damage for each heads. [FF] Swift Dive (100) If this Pokemon's remaining HP is 50 or less, this attack's base damage is 50.
Plasma Blast 55 Darkness 60 Basic Fx2 P-20 1 Common  Roar - Your opponent switches the Defending Pokemon with 1 of his or her Benched Pokemon. [1D] Ambush (20+) Flip a coin. If heads, this attack does 10 more damage.
Plasma Blast 56 Darkness 100 Stage 1 Fx2 P-20 1 Rare [1D] Dark Clamp (30) The Defending Pokemon can't retreat during your opponent's next turn. [2D] Blazing Claws (60+) If the Defending Pokemon is a Team Plasma Pokemon, this attack does 60 more damage, and the Defending Pokemon is now Burned.
Plasma Blast 60 Metal 90 Basic Rx2 P-20 1 Ultra-Rare Stellar Guidance - When you play this Pokemon from your hand onto your Bench, you may search your deck for a Supporter card, reveal it, and put it into your hand. Shuffle your deck afterward. (Ability) [2M] Hypnostrike (60) Both this Pokemon and the Defending Pokemon are now Asleep.
Plasma Blast 64 Dragon 150 Stage 2 Ox2 4 Rare Breakwing - When you play this Pokemon from your hand to evolve 1 of your Pokemon, you may discard all Pokemon Tool cards attached to each of your opponent's Pokemon. (Ability) [2RW] Gaia Crush (100) Discard any Stadium in Play.
Plasma Blast 66 Dragon 180 Basic Ox2 2 Ultra-Rare  Strafe (50) You may switch this Pokemon with 1 of your Benched Pokemon. [2GW] Dimension Heal (80) Heal from this Pokemon 20 damage for each Plasma Energy attached to this Pokemon.
Plasma Blast 70 Dragon 100 Basic Ox2 1 Common [R] Big Swing (40) Flip 2 coins. If either of them are tails, this attack does nothing. [2W] Shred (60) This attacks damage isn't affected by any effects on the Defending Pokemon.
Plasma Blast 71 Colorless 100 Basic Fx2 2 Common  Call for Family - Search your deck for 2 Basic Pokemon and put them onto your Bench. Shuffle your deck afterward.  Comet Punch (20x) Flip 4 coins. This attack does 20 damage times the number of heads.
Plasma Blast 73 Colorless 80 Stage 1 Fx2 1 Uncommon  Destructive Beam (30) Flip a coin. If heads, discard an Energy attached to the Defending Pokemon.
Plasma Blast 74 Colorless 130 Stage 2 Fx2 1 Rare Plasma Transfer - As often as you like during your turn (before your attack), you may move a Plasma Energy attached to 1 of your Pokemon to another of your Pokemon. (Ability)  Tri Attack (50x) Flip 3 coins. This attack does 50 damage times the number of heads.
Plasma Blast 75 Colorless 70 Basic Fx2 2 Common  Fury Swipes (10x) Flip 3 coins. This attack does 10 damage times the number of heads.
Plasma Blast 78 Trainer Supporter Uncommon You may play only 1 Supporter card per turn (before your attack). Put as many cards from your hand as you like on the bottom of your deck in any order. Then, draw a card for each card you put on the bottom of your deck.
Plasma Blast 79 Trainer Item Uncommon You may play as many Item cards as you like during your turn (before your attack). Look at the bottom 7 cards of your deck. You may reveal a Tirtouga you find there and put it onto your Bench. Shuffle the other cards back into your deck.
Plasma Blast 80 Trainer Item Uncommon You may play as many Item cards as you like during your turn (before your attack). Put 2 basic Energy cards from your discard pile into your hand.
Plasma Blast 81 Trainer Supporter Uncommon You may play only 1 Supporter card per turn (before your attack). During this turn, your Pokemon's attacks do 10 more damage to the Active Pokemon for each Prize card your opponent has taken (before applying Weakness and Resistance).
Plasma Blast 82 Trainer Item Uncommon You may play as many Item cards as you like during your turn (before your attack). Look at the bottom 7 cards of your deck. You may reveal an Archen you find there and put it onto your Bench. Shuffle the other cards back into your deck.
Plasma Blast 83 Trainer Item Uncommon You may play as many Item cards as you like during your turn (before your attack). Switch your opponent's Active Pokemon with 1 of his or her Benched Pokemon.
Plasma Blast 84 Trainer Supporter Uncommon You may play only 1 Supporter card per turn (before your attack). Discard your hand and draw 7 cards.
Plasma Blast 85 Trainer Item Uncommon You may play as many Item cards as you like during your turn (before your attack). Choose 1 of your Basic Pokemon in play. If you have a Stage 2 card that evolves from that Pokemon in your hand, put that card on the Basic Pokemon. (This counts as evolving that Pokemon.) You can't use this card during your first turn or on a Basic Pokemon that was put into play this turn.
Plasma Blast 86 Trainer Tool Uncommon Attach a Pokemon Tool to 1 of your Pokemon that doesn't already have a Pokemon Tool attached to it. When the Team Plasma Pokemon this card is attached to is Knocked Out by damage from an opponent's attack, search your deck for a card and put it into your hand. Shuffle your deck afterward.
Plasma Blast 87 Trainer Item Uncommon You may play as many Item cards as you like during your turn (before your attack). Look at the bottom 7 cards of your deck. You may reveal a Lileep you find there and put it onto your Bench. Shuffle the other cards back into your deck.
Plasma Blast 88 Trainer Tool Uncommon Attach a Pokemon Tool to 1 of your Pokemon that doesn't already have a Pokemon Tool attached to it. The attacks of the Pokemon this card is attached to (excluding Pokemon-EX) do 30 more damage to Active Pokemon-EX (before applying Weakness and Resistance).
Plasma Blast 89 Trainer Tool Uncommon Attach a Pokemon Tool to 1 of your Pokemon that doesn't already have a Pokemon Tool attached to it. Prevent all effects of attacks, including damage, done to the Pokemon this card is attached to (excluding Pokemon-EX) by your opponent's Team Plasma Pokemon.
Plasma Blast 90 Trainer Item Uncommon You may play as many Item cards as you like during your turn (before your attack). Discard 2 cards from your hand. (If you can't discard 2 cards, you can't play this card.) Search your deck for a Pokemon, reveal it, and put it into your hand. Shuffle your deck afterward.
Plasma Blast 91 Energy Special Energy Uncommon This card provides 1 Energy.
Plasma Blast 92 Trainer Tool Ultra-Rare Attach a Pokemon Tool to 1 of your Pokemon that doesn't already have a Pokemon Tool attached to it. You can't have more than 1 ACE SPEC card in your deck. The Genesect-EX this card is attached to can also use the attack on this card. (You still need the necessary energy to use this attack.) [1GG] G Booster (200) Discard 2 Energy attached to this Pokemon. This attack's damage isn't affected by any effects on the Defending Pokemon.
Plasma Blast 94 Trainer Item Ultra-Rare You may play as many Item cards as you like during your turn (before your attack). You can't have more than 1 ACE SPEC card in your deck. Search your deck for a Pokemon, reveal it, and put it into your hand. Shuffle your deck afterward.
Plasma Blast 95 Trainer Item Ultra-Rare You may play as many Item cards as you like during your turn (before your attack). You can't have more than 1 ACE SPEC card in your deck. Put 1 of your Pokemon and all cards attached to it into your hand.
Plasma Blast 96 Grass 170 Basic Rx2 W-20 1 Ultra-Rare Verdant Wind - Each of your Pokemon that has any G Energy attached to it can't be affected by any Special Conditions. (Remove any Special Conditions affecting those Pokemon.) (Ability) [1G] Emerald Slash (50) You may search your deck for 2 G Energy cards and attach them to 1 of your Benched Pokemon. Shuffle your deck afterward.
Plasma Blast 98 Metal 90 Basic Rx2 P-20 1 Ultra-Rare Stellar Guidance - When you play this Pokemon from your hand onto your Bench, you may search your deck for a Supporter card, reveal it, and put it into your hand. Shuffle your deck afterward. (Ability) [2M] Hypnostrike (60) Both this Pokemon and the Defending Pokemon are now Asleep.
Plasma Blast 100 Dragon 180 Basic Ox2 2 Ultra-Rare  Strafe (50) You may switch this Pokemon with 1 of your Benched Pokemon. [2GW] Dimension Heal (80) Heal from this Pokemon 20 damage for each Plasma Energy attached to this Pokemon.
Plasma Blast 101 Trainer Supporter Ultra-Rare You may play only 1 Supporter card per turn (before your attack). During this turn, your Pokemon's attacks do 10 more damage to the Active Pokemon for each Prize card your opponent has taken (before applying Weakness and Resistance).
Plasma Blast 103 Grass 110 Basic Rx2 W-20 1 Ultra-Rare  Double Draw - Draw 2 cards. [1G] Leaf Wallop (40) During your next turn, this Pokemon's Leaf Wallop attack does 40 more damage (before applying Weakness and Resistance).
Plasma Blast 105 Trainer Item Ultra-Rare You may play as many Item cards as you like during your turn (before your attack). Choose 1 of your Basic Pokemon in play. If you have a Stage 2 card that evolves from that Pokemon in your hand, put that card on the Basic Pokemon. (This counts as evolving that Pokemon.) You can't use this card during your first turn or on a Basic Pokemon that was put into play this turn.


OTTAWA, May 11 (Canadian-Media): Canada's Prime Minister, Justin Trudeau had been urged by the United Nations children's agency to press his fellow G7 leaders to help vulnerable and isolated refugee children who face sexual and physical abuse, media reports said.
The global refugee crisis, about children travelling alone without adult supervision, while in Sicily later this month had prompted a senior UNICEF official asking Trudeu to look into this matter more seriously, CTVNews reports said.
"The Canadian government and the prime minister have spoken out very strongly on two big issues: one is around refugees and migrants and secondly, is around women and girls," said UN Assistant Secretary General Justin Forsythe, the deputy executive director of UNICEF.
All G7 countries need to do more to address the exploitation of children crossing the Mediterranean Sea into Europe, Forsythe said in an interview.
Ninety percent of children making that journey were reported to be unaccompanied and faced sexual slavery and detention during their flight, and further discrimination in some European countries.
Forsythe said Trudeau had the political capital to push his fellow leaders to and help settle more unaccompanied child migrants, specially in Britain and France owing to political turmoil in these places.
The recent French election promoted anti-immigration policies of leader Marine Le Pen, who was eventually defeated.
US President, Donald Trump's attempt to ban migrants from a number of mainly Muslim countries from entering the United States had resulted in a legal battle between the White House and the U.S. judiciary.
"Even if they can't agree on the whole thing of refugees and migrants, they could agree they'll particularly look after the most vulnerable children," said Forsythe.
Forsythe said the vast majority of unaccompanied children were crossing from Libya, but children were also coming from Central American countries into the U.S.
The Mediterranean crossings recorded the most horrific circumstances, said Forsythe and siad he had met a Nigerian teenage girl who been imprisoned in an underground prison in Libya for eight months and repeatedly raped.
Teenaged boys from Eritrea described being sold into slavery to a militia of 150 children in Libya, he said.
According to the UNICEF reports, 96,000 unaccompanied children claimed asylum in Europe in 2015, but the full picture for 2016 was not yet known. However, in Germany alone, some 36,000 unaccompanied children had claimed asylum.


Today's College Girl of the Day is Jackie from Clemson University. We figured we'd give the Tigers some love after their repeat bid died against Alabama in the Sugar Bowl. Chin up, Tigers fans… it's time to get hyped (way too early) for next season.


(ID) and take 75 east about 50 miles to Challis, ID.
Follow right on the edge of the twisting Salmon River (HWY 75) in central Idaho from the top of the Sawtooth Mtns to the desert! This is a great route in one of the deepest canyons in the US. You are not riding at elevation like the Beartooth. You are at the bottom, right beside the river, and the curves and drop offs are endless. Just a great enjoyable ride!
The road is in good shape with endless curves.
Fairly quiet road when i was on it. Lots of scenery changes, from the green lush high elevations down to the desert! White water rafting is popular on this river and you will be passing them all day! Lots of campgrounds near the top and a dip in the river to cool down is possible just about anywhere you want to pull over.
This was one of my last rides for the year and will be one of my first this spring. Great riding even thou there wes a big forest fire on the North Fork of the Salmon River. smoke wasent too bad but a lot of firefighter traffic. stayed in the town of Salmon and enjoyed it and it was very inexpensive place to stay. we were lucky to get a room with all of the firefighters in town.
on November 23, 2011 1 out of 1 members found this review useful.
Rode US Hwy. 93 north from Kingman, Arizona up to Polson, Montana. Then went east around Flathead Lake up to and through Glacier National Park in July, 2009. Veered north onto State Route 75 in Shoshone, Idaho going through Ketchum and over the Sawtooth MTNS. into Stanley. Very good and memorable ride along the Salmon River. I rate it among my top five rides in the lower forty-eight.
on September 17, 2018 0 out of 0 members found this review useful.
This is an incredible ride. Very pretty and fun ride along the river and through the mountains and countryside. A nice balance of twisties and open rode. Watch for rocks and game (especially early AM and around dusk.) Try to stop at the hot springs in Stanley and don't miss Red Fish Lake. The view of the Sawtooths from Lower Stanley is like someone painted the perfect mountain backdrop on a canvas. Highly recommend! Gorgeous scenery.
on December 25, 2009 0 out of 0 members found this review useful.
2009 my wife and rode hwy 93 all the way from the Canadian border to Stanley. This is a great ride. Make sure if going north you keeping going to Hamilton and stop at the Hamilton House for some really good food.


Compare multiple Bridgeton, NC life insurance quotes in order to find the most affordable and best-quality coverage. Enter your Zip above and you will be able to choose from the best options available in Craven county. Life insurance comparison is the best way to save money on your coverage and ensure you are not overpaying.
If you want to buy Bridgeton, North Carolina life insurance you need to make sure you purchase the best policy for the best rate. Insurance can be expensive depending on the type of policy and the policy amount.
When you go online, you can find Bridgeton, NC life insurance quotes from insurance companies and brokers. The insurance company would be the issuing company. They will generally give you a quote for their policies. Insurance brokers and agents can provide you with a number of quotes, depending on the companies they work with.
Prices are climbing on almost almost everything across the globe, from hire, gas and groceries, all the way up to the expense of a funeral. You should not go away your family burdened monetarily, should you move. Use the suggestions in this article to find and purchase a existence insurance coverage plan, so that the folks you treatment most about are secured.
Make confident to get rates on distinct levels of guidelines. Numerous insurance coverage organizations supply breaks at diverse amounts of protection that could wind up saving you cash. Just simply because you've determined that a hundred seventy five,000 is all the protection you need, doesn't indicate you shouldn't get quoted on other ranges just in circumstance.
When deciding on daily life insurance coverage, be sure to have a great concept of how considerably funds your cherished kinds will call for. This is critical simply because of the volume of financial debt that they might be still left with the load of paying off after you die. Contemplate auto payments, mortgage, credit history card debts, funeral and burial fees, and the price of boosting youngsters.
An insurance policies program is not a get-it and overlook-it buy. You will want to revise your insurance policies program as factors alter in your daily life and you get older. Modifications in marital status, obtaining youngsters, or reaching retirement age are all causes to overview your strategy and make changes.
Investigation your daily life insurance policies policy extensively before you indicator on the dotted line. Make certain that you are conscious of the exclusions, inclusions, reductions, no payment for dying induced by a pre-current condition clauses and other phrases of your life insurance coverage coverage just before committing to it. With out understanding specifically what your policy handles, when you die you could inadvertently leave household customers still left behind in a tough position.
Once you have established you require lifestyle insurance policy, next you want to figure out how much you need. There are numerous on the internet calculators you can use, and some other formulation an insurance policy agent can help you with. If you want a ballpark determine quickly, just take your wage and multiply it by eight. This will give you an estimate of how much life insurance policy you need.
Often a existence insurance coverage holder wishes to modify their beneficiary following their plan is currently set. Don't fret about this. The firm will not penalize you for a change. You just have to make certain that you fill out the correct varieties in the presence of a witness in buy to finalize the alterations.
Try to purchase your existence insurance policies coverage as shortly as you need it. It is certainly ideal to do this when you are younger and healthy due to the fact the policy's rates tend to be much cheaper. If you hold out until later on when you are more mature or when you are in inadequate wellness, the premiums can be quite pricey.
Getting term lifestyle insurance policies, as opposed to total-lifestyle insurance, is a sensible selection for most customers, but picking the appropriate time period length is key. Factors to take into account as you choose the expression is your personal age, the age of your dependents, the character of your financial commitments, as effectively as what you can reasonably find the money for. You might want to think about basing the expression close to achievement of milestone bills like when your youngest youngster will have graduated from university or when the home will be entirely compensated off. Alternatively, numerous people decide on a phrase that addresses them till they can access their retirement sources. What ever your personal considerations may be, choosing your phrase size thoughtfully will provide numerous several years of peace of mind.
From being aware of how to save income on daily life insurance coverage to figuring out which sort of plan you must go with, this article has explained some of the finer details of the insurance policies industry, so that you can uncover the best offer feasible on a high quality insurance coverage strategy. Act before it is way too late. Choose up that program.


They're terrified a mass credit freeze will crush revenues.
Shares of Equifax dropped another 4% today, including after-hours, to $92.70. They're now down 35%, or $50, from the happier era that ended at 5pm EST on September 7, with the confession that it had found out six weeks earlier that the most crucial personal data – "primarily names, Social Security numbers, birth dates, addresses and, in some instances, driver's license numbers" – of 143 million consumers had been stolen.
This was promptly followed by chaos and egregious missteps, such as trying to profit from its victims. So far, at 120.4 million shares outstanding as of June 30, the six trading days have cost investors $6 billion. No one cares about consumers. They're just the product. But $6 billion matter.
Much was made of Mauldin's degrees in music. But for a person her age, and with as much corporate experience as she had, college is irrelevant. Gates, Jobs, and Zuckerberg didn't even graduate from college. What matters is how they perform their work.
And they failed to patch a vulnerability in Apache Struts, an open-source and therefore free software. The vulnerability had been "identified in early March" but wasn't patched. The hack occurred from May 13 through July 30, 2017.
While Equifax fully understands the intense focus on patching efforts, the company's review of the facts is still ongoing.


They are renowned for their cheerful, humorous announcements, which have seen Greater Anglia's social media channels flooded with praise, and this year they recorded their lowest ever levels of absence.
Now Southend Victoria's driver team has been crowned Team of the Year at the train company's 'Everyday Legends' Awards.
Greater Anglia's driver team is made up of two driver managers and 81 drivers who operate the trains between Southend Victoria and London Liverpool Street. The team likes to bring some personality to their announcements to help passengers feel welcome and relaxed during their journey.
Greater Anglia's Managing Director, Jamie Burles, said, "Everyday, colleagues across the network do their utmost to provide excellent customer service and make our passengers' lives a little easier. I would like to congratulate and thank not only the winners and nominees but everyone across the company who works hard every day to deliver excellent service and pleasant journeys to our passengers."


text:
Advances in Artificial Pancreas Development
Quantitative Estimation of Insulin Sensitivity in Type 1 Diabetic Subjects Wearing a Sensor-Augmented Insulin Pump
Michele Schiavon1,
Chiara Dalla Man1,
Yogish C. Kudva2,
Ananda Basu2 and
Claudio Cobelli1⇑
1Department of Information Engineering, University of Padova, Padova, Italy
2Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Internal Medicine, Mayo Clinic and Foundation, Rochester, MN
Corresponding author: Claudio Cobelli, cobelli{at}dei.unipd.it.
Diabetes Care 2014 May; 37(5): 1216-1223. https://doi.org/10.2337/dc13-1120
OBJECTIVE The goal was to develop a new index of insulin sensitivity in patients with type 1 diabetes estimated from continuous glucose monitoring (CGM) and subcutaneous insulin delivery data under carefully controlled conditions.
RESEARCH DESIGN AND METHODS The database consists of 12 subjects with type 1 diabetes, studied during breakfast, lunch, and dinner, in a clinical research unit, wearing both subcutaneous insulin pump and CGM device. Frequent blood samples were drawn for measurements of plasma glucose and insulin concentrations in order to estimate insulin sensitivity with the oral minimal model (SIMM). The new index of insulin sensitivity (SISP) was calculated with a simple algebraic formula for each meal, using only CGM and insulin pump data and compared with SIMM.
RESULTS SISP was well correlated with SIMM (r = 0.825; P < 10−8), and diurnal pattern was also similar to SIMM.
CONCLUSIONS A novel method for estimating insulin sensitivity in subjects with type 1 diabetes on sensor-augmented insulin pump therapy has been presented. This new index correlates well with the reference oral minimal model estimate of insulin sensitivity. The knowledge of patient-specific insulin sensitivity and its diurnal variation can help in optimizing insulin therapy in type 1 diabetes and could also inform next-generation closed-loop control systems.
The standard therapy for type 1 diabetes consists of exogenous insulin administration, either by multiple daily injections or with continuous subcutaneous insulin infusion (CSII) through the insulin pump, adjusted according to self-monitored blood glucose (SMBG) levels 3–4 times per day. However, in the past 10–15 years, new possibilities in diabetes therapy have emerged thanks to continuous glucose monitoring (CGM) and CSII, which substitute SMBG and multiple daily injection therapy, respectively. Minimally invasive CGM devices can measure, in real time, interstitial glucose (IG) concentrations in continuous time for up to several days. CSII uses the subcutaneous route and administers insulin with a basal/bolus strategy, i.e., continuously deliver insulin over 24 h and inject boluses.
In order to determine the appropriate meal insulin bolus, it would be important to know the subject-specific insulin sensitivity, i.e., the ability of insulin to stimulate glucose utilization and inhibit glucose production. Indices of insulin sensitivity include those based on hyperinsulinemic-euglycemic clamp (1), intravenous glucose tolerance test (2), and, more recently, meal and oral glucose tolerance test. These indices include the oral glucose minimal model (3,4), a minimal model based integral formula (5), and surrogate measures (6–8). However, all the above require the measurement of both plasma glucose and insulin concentrations and therefore cannot be used in everyday life. In the outpatient setting, the only usable approach is the risk method proposed by Breton and Kovatchev (9), which uses SMBG data collected over a period of 2–6 weeks and some patient parameters. This index measures the average insulin sensitivity in the preceding 2 weeks and thus cannot be used to assess the intraday variability of insulin sensitivity.
Here we propose a method to estimate insulin sensitivity from CGM sensor and insulin pump data and validate it against the oral minimal model, which uses plasma glucose and insulin concentrations. The method is usable in everyday life in patients wearing the two devices.
Database and Protocol
Twelve type 1 diabetic subjects (5 females, aged 39.5 ± 14.2 years, BMI 25.7 ± 3.8 kg/m2, HbA1c ≤8.5% or 69 mmol/mol) were studied for 3 days in the clinical research unit of the Mayo Clinic Center for Translational Science Activities as part of a separate study to determine diurnal patterns of insulin sensitivity (10). Briefly, once per day, a triple-tracer mixed-meal study protocol was performed during breakfast, lunch, or dinner in Latin square design. Blood samples were collected at −180, −30, 0, 5, 10, 20, 30, 60, 90, 120, 150, 180, 240, 300, and 360, with t = 0 corresponding to meal time, for measurement of plasma glucose and insulin concentrations in order to estimate SI with the oral minimal model (3), here considered as reference. More details can be found in Hinshaw et al. (10). Subjects also wore both subcutaneous insulin pump (Medtronic or Insulet OmniPod) and CGM (Dexcom Seven Plus). Within-subject mean absolute relative difference between CGM readings and reference was equal to 12.1 ± 3.3%, and no systematic variation over time was found (mean [SD] within subjects was 5.3 ± 3.7%). This is perfectly in line with published reports (11). Figure 1 shows CGM and insulin infusion rate in a representative subject. The 12 subjects have been chosen among the 19 reported by Hinshaw et al. (10) since they had the complete CGM and CSII data required for the calculation described below. Of note, one CGM trace was missing in one subject during one meal.
Data needed for the estimation of insulin sensitivity. Glucose data are measured by CGM (left-hand panel, black line), with some SMBG data available for sensor recalibration (left-hand panel, gray dots). Insulin data are the basal infusion and boluses administered by insulin pump (right-hand panel, black line and gray triangles, respectively).
The starting point is the derivation of insulin sensitivity by integrating the oral minimal model equations (3), as described by Caumo et al. (5). Then, the calculation is adapted to allow the use of CGM and CSII data, instead of plasma glucose and insulin concentrations. The oral minimal model (3) is
where G is plasma glucose concentration (milligrams per deciliter), with Gb denoting its basal value; X is insulin action (min−1); I is plasma insulin concentration (microunits per milliliter), with Ib denoting its basal value; RaG is posthepatic appearance of meal glucose (milligrams per kilograms per minute); VG is glucose distribution volume (deciliters per kilogram); p2 is speed of rise and decay of insulin action (min−1); and p3 is its size (min−2 per microunit per milliliter). Insulin sensitivity is defined as
By substituting
into Eq. 1, we obtain
By integrating the differential equation (Eq. 3) from the time of the meal ingestion (tmeal) to the end of the experiment (tend) and rearranging, one has
The first integral in the numerator can be rewritten as
where D (milligrams) is the amount of glucose ingested during the meal and f(tend) is the fraction of the ingested dose, which at t = tend has reached plasma. X′ in Eq. 4 is usually not available because model parameters are unknown, thus the denominator of Eq. 4 is substituted with the average of glucose excursion times the overall insulin stimulus:
where |ΔG| is above basal plasma glucose concentration.
Thus SI is given by
where AUC is the area under the curve calculated from the start of the meal (tmeal) to the end of the experiment (tend). Subject-specific parameters used by the formula are the body weight (BW) (kilograms), height (meters), and age (years). BW is explicitly used in Eq. 7 and, together with height and age, used for the calculation of plasma insulin clearance (CL), as discussed below. Parameter fixed to population values are glucose effectiveness at zero insulin GEZI [deciliters per kilogram per minute; fixed to 0.01 dL/kg/min for diabetic subjects (10,12)] and volume of glucose distribution VG [deciliters per kilogram; fixed to 1.45 dL/kg, according to Dalla Man et al. (3)].
G, ΔG, and I are not directly available but can be derived from CGM and subcutaneous insulin delivery as detailed below.
Glucose Signal From CGM
CGM measures the IG concentration, which is related to plasma glucose by a first-order differential equation, i.e., IG is a delayed version of plasma glucose:
Thus, with a well-calibrated device, assuming IG(tend) = IG(tmeal), one has
i.e.,
Similarly, for the above basal glucose signal, one has
Thus one can safely assume that, in the presence of a well-calibrated device, AUC(ΔCGM) and AUC(|ΔCGM|) are good approximations of AUC(ΔG) and AUC(|ΔG|), respectively.
In presence of a noncalibrated device, if at least two SMBG samples are available, it is possible to offline recalibrate the CGM profile (13,14).
Insulin Signal From CSII
For the calculation of AUC(I), we assume that insulin is not degraded locally in the site of infusion. In this case, the integral of plasma insulin can be obtained from subcutaneous insulin infusion divided by the plasma insulin clearance CL. In fact, plasma insulin kinetics can be described with a single compartment model (15):
where I(t) is the plasma insulin concentration, RaI(t) is the insulin rate of appearance in plasma, VI is the insulin volume of distribution, and n the fractional insulin clearance rate (n = CL/VI).
Then, integrating Eq. 12 from the time of the premeal bolus (tmeal) to the end of the observation period (tend) and assuming that insulin is back to its initial value at the end of the experiment, one has
Finally, under the assumption that all infused insulin eventually reaches the circulation, one has
Thus we can compute AUC(I) from the amount of insulin infused subcutaneously and CL:
where basal(t) is the basal insulin infusion rate during the integration period, bolus(tk) is the premeal or correction bolus administered at t = tk, and CL (liters per minute) is the plasma insulin clearance, which can be calculated from subject's height, BW, and age by using the population model proposed by Campioni et al. (16). In addition, if correction boluses are administered before the start of the meal, one has to consider that part of that injected insulin could be still active. The residual active insulin can be determined by adopting the same algorithm presented by Patek et al. (17), which uses the insulin on board (IOB) curves adapted from Ellingsen et al. (18). This quantity (IOB[tmeal]) must be added to the previously estimated AUC(I). Moreover, if correction boluses are administered before the end of the considered interval, IOB is used to evaluate the active insulin at the end of the study (IOB[tend]), which is subtracted from the previously estimated AUC(I). In summary,
Accounting for Carbohydrates on Board
As evident from Eq. 7, an accurate calculation of insulin sensitivity requires the knowledge of the amount of carbohydrates entering the circulation in the integration interval [D·f(tend)]. In this study, meals were provided at 7:00 a.m., 1:00 p.m.; and 7:00 p.m. each day, thus the time interval between the meals was at least 6 h. However, this may not always be the case. Furthermore, not all the carbohydrates ingested with a meal may be fully absorbed before the ingestion of a second meal. Thus, to account for unabsorbed carbohydrates, e.g., frequent meals close to each other, the concept of carbohydrates on board (COB) is introduced. Similarly to IOB, COB is a function that, at each time t, quantifies the fraction of the ingested carbohydrates that has not yet appeared in the circulation. COB is based on the model of gastrointestinal tract (19): given the amount of carbohydrates ingested at time tm, COB provides, at each time t > tm, the percentage of carbohydrates not yet absorbed, while for t–tm >360 min, it is assumed that the carbohydrate absorption is almost completed. The fraction of the ingested dose that has reached plasma at time t, f(t), can be calculated as the ratio between the AUC of the meal rate of appearance and the ingested dose, D (Fig. 2, left panel), assuming that, at the end of the meal, the fraction of the meal appearing in plasma is f∞ = 0.9 (Fig. 2, right panel). The time course of f is shown in Fig. 2, right panel. COB is then calculated as COB(t) = f∞–f(t).
Time course of meal rate of appearance, RaG (left-hand panel), and fraction of meal that appears in plasma, f (right-hand panel).
To better grasp the use of f(t) and COB(t) for the calculation of insulin sensitivity, let us consider the following example: suppose that a subject eats 50 g of carbohydrates at t = 0 min and another 40 g at t = 180 min then fasts for more than 360 min. One can calculate two values of SI, one for each meal ingestion (SImeal1 and SImeal2). For the calculation of SImeal1, the amount of ingested glucose to be considered is 50·f(180) g of carbohydrates, while for the calculation of the second, SImeal2, one should use 50·COB(180)+40·f∞.
Generalizing, the amount of carbohydrates to use in the formula for the ith meal is
where AoC is the amount of carbohydrates, D(tmeali) is the amount of carbohydrates ingested at the time of the ith meal (tmeali), tendi is the time at which the ith meal ends (corresponding to the time of ingestion of the [i+1]th meal), and COB(tendi–1) the COB at the end of the (i–1)th meal, which contained D(tmeali–1) amount of carbohydrates.
Insulin Sensitivity From Sensor and Pump Data
Incorporating the above derivations into Eq. 7, one can estimate insulin sensitivity from sensor and pump data (SISP) for the ith meal (20):
It is important to define the domain of validity of Eq. 18. When CGM is still high 6 h after meal ingestion, SISP can become negative. This brings the method working outside its domain of validity. We thus recommend to use the formula only if the recalibrated ΔCGM is lower than 150 mg/dL 6 h after meal ingestion. As a matter of fact, one of our subjects (subject 4) during lunch has recalibrated ΔCGM greater than 150 mg/dL 6 h after meal ingestion, and SISP was negative. This subject was thus excluded in the following analysis.
Minimal Model Insulin Sensitivity and Validation of SISP
The oral glucose minimal model (3,4) was used to estimate insulin sensitivity, from plasma glucose and insulin concentrations, in 12 subjects studied three times (10). Here we consider these measures as reference values (SIMM) to which SISP is compared for validation.
Assessment in Case of Incompletely Absorbed Meals
In our experimental protocol, meals were well spaced (at least 6 h) and thus completely absorbed before the next meal. However, in daily life, this may not happen consistently. Thus it is important to assess the performance of the method in case of incompletely absorbed meals. We did so by comparing estimates of SISP obtained with a 360-min interval to those obtained from shorter intervals (up to 180 min), both with and without using the COB function.
When proposing a new metric like SISP, it is important to assess its reproducibility, i.e., if it provides similar values when repeatedly applied to the same subject under the same experimental conditions. However, due to the large intrasubject variability of insulin sensitivity (10), SISP will likely change if the same meal is administered in two occasions to a subject. Thus, to address reproducibility, one can resort to simulation. We evaluated SISP reproducibility in simulation using our U.S. Food and Drug Administration-accepted type 1 diabetes simulator (21). We simulated a 7-day scenario for 100 in silico subjects with three meals per day (breakfast from 6:00 to 8:00 a.m., lunch from 11:30 a.m. to 1:30 p.m., and dinner from 6:00 to 8.30 p.m.) with different amounts (breakfast 0.7 – 0.9 g/kg, lunch 0.8 – 1.0 g/kg, and dinner 0.8 – 1.3 g/kg), while subject-specific insulin sensitivity was maintained constant for the whole simulation. A total of 21 SISP were thus calculated for each subject (three values per 7 days). To assess the repeatability of the index, we calculated the average (mean), the SD, and the coefficient of variation (CV; SD/mean) of the 21 estimates of SISP in each subject.
Data are presented as mean ± SD. Two sample comparisons were done by paired sample t test. Pearson's correlation was used to evaluate univariate correlation.
The correlation between the two indices was very good (r = 0.825; P < 10−8; Fig. 3, right-hand panel), and diurnal pattern was similar, indicating that, apart from a scale factor, SISP closely mirrors SIMM. SIMM and SISP have been estimated in the 12 subjects at breakfast, lunch, and dinner. SISP was significantly higher than SIMM (13.86 ± 14.56 vs. 6.67 ± 5.63 dL/kg/min per μU/mL; P < 10−3; Fig. 3, left-hand panel).
Mean values (left-hand panel) and correlation plot (right-hand panel) between SIMM and SISP insulin sensitivity indices for breakfast, lunch, and dinner (white, striped, and black bars, respectively). *P < 10−3 with paired sample t test.
When SISP was calculated for incompletely absorbed meals, i.e., relying on reduced integration intervals, mean values of SISP were virtually the same (Fig. 4, top left) if one uses the COB function. Conversely, SISP increased systematically if COB is not used (Fig. 4, top right). Of note, the correlation between SISP calculated at the end of the experiment and that obtained from reduced integration intervals decreases only slightly (Fig. 4, middle panels), and the absolute relative error increases (Fig. 4, bottom panels) both with and without using COB. Finally, in silico reproducibility of SISP was 23 ± 6%.
Sensitivity analysis of SISP with (left-hand panel) and without (right-hand panel) accounting for COB for different time integration intervals: mean values of SISP (top), correlation indices (middle), and absolute relative error (bottom), calculated with respect to SISP, i.e., estimated at t = 360 min.
Insulin sensitivity is a key parameter of the metabolic status of an individual, which could be beneficial also for optimizing insulin therapy in type 1 diabetes. In fact, the knowledge of patient-specific SI and its daily variation can truly help in determining the optimal insulin bolus to be administered to cover the ingested carbohydrates. However, all methods available for the estimation of insulin sensitivity rely on plasma glucose and insulin measurements and thus cannot be used in everyday life of a patient with type 1 diabetes.
In this article, we have proposed an index of insulin sensitivity, SISP, which can be estimated in patients with type 1 diabetes wearing a CGM sensor and an insulin pump. We have demonstrated that it is similar to the one obtained with the oral minimal model, which requires plasma glucose and insulin data. The method uses retrospective subcutaneous sensor and insulin delivery data with some anthropometric parameters for each subject and provides, for each meal, the patient's insulin sensitivity by an integral formula.
SISP measures how subcutaneously infused insulin affects the CGM profile. Thus SISP is not exactly the same index derived with the minimal model (SIMM), which represents the ability of insulin to suppress endogenous glucose production and stimulate glucose uptake. In other words, SISP is a new metric of insulin sensitivity with its own range. In fact, mean values of SISP were almost twice SIMM. However, the correlation between the two indices was excellent (r = 0.825; P < 10−8).
A robust estimate of insulin sensitivity can be obtained for each meal whenever meals are well spaced (5–6 h). However, we also tested the method in case of incompletely absorbed meals. To deal with this situation, we introduced the concept of COB. Similarly to IOB (18), COB represents, at each time t, the amount of ingested glucose that has not yet been absorbed. Thus one can define different COB curves for different types of meal, i.e., fast or slow carbohydrates. We used COB in the SISP calculation to evaluate the correct amount of carbohydrates in relation with the observed CGM profiles in a given time interval. We demonstrated the need of using COB; in case of a short time interval between consecutive meals, a robust estimation of SISP can only be obtained if carbohydrates absorption is taken into account (Fig. 4, top left-hand versus top right-hand panel).
Possible applications of the new index include its use for assessing intraday and interday variability (e.g., existence of diurnal patterns) of insulin sensitivity in large cohorts of subjects with type 1 diabetes in normal life conditions. For instance, the method is currently being applied to the Sensor-Augmented Pump Therapy for A1C Reduction (STAR) 3 data of type 1 diabetic subjects wearing a sensor-augmented insulin pump (22). This will provide, in each subject, the SISP time course during several months and will allow testing for the existence of subject-specific SISP daily patterns and correlation of such variation with lifestyle and other factors. Following validation using STAR 3 data, a clinical application would be the use of SISP to calculate the optimal insulin-to-carbohydrate ratio (CR) (23,24). Thus optimizing CR based on recent CGM and CSII data collected 1–2 weeks prior to a clinical visit may be useful for physicians to improve patient-specific meal bolus insulin therapy and thus a major component of glucose control. In addition, the knowledge of patient CR daily pattern may help in the design of optimal closed-loop control algorithms relying on patient-specific open-loop insulin therapy at the time of transition from open- to closed-loop therapy. A natural progression of this work would be the development of real-time adaptation of open- or closed-loop therapy based on the presence of well-developed metrics such as the one developed here. Further development of diabetes technology hardware and algorithms should make this a reality over the next several years.
The method relies on data provided by a CGM device that can occasionally suffer from inaccuracies. For instance, the assumption that AUC(CGM) is a good approximation of AUC(G) becomes critical if the device is not well calibrated. If this occurs, SISP will also reflect sensor inaccuracy. However, to improve the quality of CGM measurements, some algorithms can be used to recalibrate CGM traces (13,14). Another possible limitation is the need to fix some parameters (GEZI and VG) to population values (3,10,12) and others calculated from population models (CL) using anthropometric data (16). In order to test the effect of fixing these parameters, we also calculated SISP using individualized GEZI, estimated with the oral glucose minimal model (3) and CL, directly estimated from the data in each patient; we obtained values very similar to those obtained with fixed parameters and a slightly higher correlation with SIMM.
Insulin sensitivity is an important element in the daily life of patients with type 1 diabetes and could be useful to optimize insulin therapy. However, methods to estimate this index by emerging technologies, such as subcutaneous CGM sensor and insulin pump, has never been proposed. We have presented a method that estimates insulin sensitivity from CGM and insulin pump data. Future studies involving a larger databases that include larger cohorts of subjects studied for a longer time are needed to better define the applicability in free-living conditions.
Acknowledgments. The algorithm proposed in this article is part of a patent request deposited by the University of Padova (13/661,755).
Funding. The work was partially supported by the National Institutes of Health (DK-R01-085561 and DK-DP3-094331); grant number UL1-TR000135 from the National Center for Advancing Translational Sciences, a component of the National Institutes of Health; and the Italian Ministero dell'Istruzione, dell'Università e della Ricerca (Progetto FIRB 2009).
Duality of Interest. No potential conflicts of interest relevant to this article were reported.
Author Contributions. M.S., C.D.M., and C.C. developed the method, analyzed the results, and drafted the manuscript. Y.C.K. and A.B. commented on and edited the manuscript. C.C. is the guarantor of this work and, as such, had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Prior Presentation. This work was presented in part at the 11th Annual Diabetes Technology Meeting, San Francisco, CA, 27–29 October 2011, and the 12th Annual Diabetes Technology Meeting, Bethesda, MD, 8–10 November 2012.
See accompanying articles, pp. 1182, 1184, 1191, 1198, 1204, 1212, and 1224.
Received May 10, 2013.
Accepted October 24, 2013.
© 2014 by the American Diabetes Association.
Readers may use this article as long as the work is properly cited, the use is educational and not for profit, and the work is not altered. See http://creativecommons.org/licenses/by-nc-nd/3.0/ for details.
DeFronzo RA,
Tobin JD,
Andres R
. Glucose clamp technique: a method for quantifying insulin secretion and resistance. Am J Physiol 1979;237:E214–E223pmid:382871
Bergman RN,
Ider YZ,
Bowden CR,
Cobelli C
. Quantitative estimation of insulin sensitivity. Am J Physiol 1979;236:E667–E677pmid:443421
Dalla Man C,
Caumo A,
Basu R,
Rizza R,
Toffolo G,
. Minimal model estimation of glucose absorption and insulin sensitivity from oral test: validation with a tracer method. Am J Physiol Endocrinol Metab 2004;287:E637–E643pmid:15138152
Yarasheski KE,
. Insulin sensitivity by oral glucose minimal models: validation against clamp. Am J Physiol Endocrinol Metab 2005;289:E954–E959pmid:16014353
. Insulin sensitivity from meal tolerance tests in normal subjects: a minimal model index. J Clin Endocrinol Metab 2000;85:4396–4402pmid:11095485
Stumvoll M,
Mitrakou A,
Pimenta W,
. Use of the oral glucose tolerance test to assess insulin release and insulin sensitivity. Diabetes Care 2000;23:295–301pmid:10868854
Matsuda M,
DeFronzo RA
. Insulin sensitivity indices obtained from oral glucose tolerance testing: comparison with the euglycemic insulin clamp. Diabetes Care 1999;22:1462–1470pmid:10480510
Hansen T,
Drivsholm T,
Urhammer SA,
. The BIGTT test: a novel test for simultaneous measurement of pancreatic β-cell function, insulin sensitivity, and glucose tolerance. Diabetes Care 2007;30:257–262pmid:17259491
Breton MD, Kovatchev BP. Method, system and computer program product for evaluation of insulin sensitivity, insulin/carbohydrate ratio, and insulin correction factors in diabetes from self-monitoring data. PCT international patent application PCT/US2008/069416. 8 July 2008
Hinshaw L, Dalla Man C, Nandy DK, et al. Diurnal pattern of insulin action in type 1 diabetes: implications for a closed-loop system. Diabetes 2013;62:2223–2229
Leelarathna L,
Nodale M,
Allen JM,
. Evaluating the accuracy and large inaccuracy of two continuous glucose monitoring systems. Diabetes Technol Ther 2013;15:143–149pmid:23256605
Basu A,
Cobelli C,
Rizza RA
. Effects of type 2 diabetes on insulin secretion, insulin action, glucose effectiveness, and postprandial glucose metabolism. Diabetes Care 2009;32:866–872pmid:22481799
Guerra S,
Facchinetti A,
Sparacino G,
Nicolao GD,
. Enhancing the accuracy of subcutaneous glucose sensors: a real-time deconvolution-based approach. IEEE Trans Biomed Eng 2012;59:1658–1669
Del Favero S,
. Retrofitting algorithm: fusing CGM and references to reconstruct a reliable continuous glucose profile. Paris, Advanced Technologies and Treatments for Diabetes, 2013pmid:16144811
Toffolo GM,
Campioni M,
Rizza RA,
. A minimal model of insulin secretion and kinetics to assess hepatic insulin extraction. Am J Physiol Endocrinol Metab 2006;290:E169–E176pmid:22481809
. Minimal model assessment of hepatic insulin extraction during an oral test from standard insulin kinetic parameters. Am J Physiol Endocrinol Metab 2009;297:E941–E948pmid:20144293
Patek SD,
Magni L,
Dassau E,
International Artificial Pancreas (iAP) Study Group
. Modular closed-loop control of diabetes. IEEE Trans Biomed Eng 2012;59:2986–2999pmid:19671837
Ellingsen C,
Zisser H,
. Safety constraints in an artificial pancreatic beta cell: an implementation of model predictive control with insulin on board. J Diabetes Sci Tech 2009;3:536–544pmid:19196896
Camilleri M,
. A system model of oral glucose absorption: validation on gold standard data. IEEE Trans Biomed Eng 2006;53:2472–2478pmid:17153204
Cobelli C, Dalla Man C, Schiavon M, Basu A, Kudva Y. Estimation of insulin sensitivity from CGM and subcutaneous insulin delivery in type 1 diabetes. U.S. patent application US 13/661,755. 26 October 2012
Kovatchev BP,
Breton M,
Man CD,
. In silico preclinical trials: a proof of concept in closed-loop control of type 1 diabetes. J Diabetes Sci Tech 2009;3:44–55pmid:19444330
Davis SN,
Horton ES,
Battelino T,
Rubin RR,
Schulman KA,
Tamborlane WV
. STAR 3 randomized controlled trial to compare sensor-augmented insulin pump therapy with multiple daily injections in the treatment of type 1 diabetes: research design, methods, and baseline characteristics of enrolled subjects. Diabetes Technol Ther 2010;12:249–255pmid:20210562
Yodfat O, Shapira G, Gescheit I. A method system and device for assessing insulin sensitivity. PCT publication US20080319381. 31 December 2008
Bevier WC,
Palerm CC,
. Calculating the insulin to carbohydrate ratio using the hyperinsulinaemic-euglycaemic clamp-a novel use for a proven technique. Diabetes Metab Res Rev 2007;23:472–478pmid:17315240
You are going to email the following Quantitative Estimation of Insulin Sensitivity in Type 1 Diabetic Subjects Wearing a Sensor-Augmented Insulin Pump
Michele Schiavon, Chiara Dalla Man, Yogish C. Kudva, Ananda Basu, Claudio Cobelli
Diabetes Care May 2014, 37 (5) 1216-1223; DOI: 10.2337/dc13-1120
Feasibility of Closed-Loop Insulin Delivery in Type 2 Diabetes: A Randomized Controlled Study
The Artificial Pancreas: Are We There Yet?
Closed-Loop Artificial Pancreas Systems: Physiological Input to Enhance Next-Generation Devices
Show more Advances in Artificial Pancreas Development

redpajama_set_name: RedPajamaCommonCrawl


text:
Jobs at TD Bank in Clinton, MD
Store Customer Experience Coordinator PT- 20 Hours (Annandale)
TD Bank - Annandale, VA
TD Description About TD Bank, America's Most Convenient Bank® TD Bank, America's Most Convenient Bank, is one of the 10 largest banks in the U.S., providing more than 8 million customers with a full range of retail, small business and commercial banking products and services at approximately 1,300 convenient locations throughout the Northeast, Mid-Atlantic, Metro D.C.,...
Careers at TD Bank
Jobs in Annandale, VA
TD Bank Jobs in Annandale, VA
CSR II (Tenleytown)
TD Bank - Washington, DC
TD Bank Jobs in Washington, DC
Customer Experience Coordinator, Rockville Pike, 20 hours per week
TD Bank - North Bethesda, MD
Jobs in North Bethesda, MD
TD Bank Jobs in North Bethesda, MD
CSR II (McLean)
TD Bank - Falls Church, VA
Jobs in Falls Church, VA
TD Bank Jobs in Falls Church, VA
Store Customer Experience Coordinator- PT 20 Hrs (Georgetown)
Head Teller, Beltsville, Full Time
TD Bank - Beltsville, MD
Jobs in Beltsville, MD
TD Bank Jobs in Beltsville, MD
Head Teller – LT, Silver Spring
TD Bank - Silver Spring, MD
Jobs in Silver Spring, MD
TD Bank Jobs in Silver Spring, MD
Community Mortgage Sales Manager (US)
TD Description About TD Bank, America's Most Convenient Bank® TD Bank, America's Most Convenient Bank, is one of the 10 largest banks in the U.S., providing more than 8 million customers with a full range of retail, small business and commercial banking produc Job Description Job Description Summary The Community Mortgage Sales Manager m...
Store Customer Experience Coordinator- PT 30 Hrs ( Beacon Hill)
TD Bank - Alexandria, VA
TD Bank Jobs in Alexandria, VA
Assistant Store Manager (Seven Corners)
Assistant Store Manager (Beacon Hill)
Sr Relationship Manager
TD Bank - Vienna, VA
Jobs in Vienna, VA
TD Bank Jobs in Vienna, VA


The Gleam system of accent tables is a modular system consisting of wood or marble tops available in a variety of different finishes and colors. Paired with elegant metal legs that are either lacquered black or chromed, there are dozens of different combinations available. Available in a square or rectangular coffee table and in a side table.
Dimensions: 16 x 30 x 30"


Do Boys Just Wanna Have Fun?
Brad first met Natalie on a MOO. He was a college senior at an eastern university, she a junior on the west coast. They got to know each other better by corresponding through e-mail.
브래드는 나탈리를 MOO에서 만났다. 그는 동부 대학을 다니는 고학년 학부생이고, 그녀는 서부 해안 대학의 저학년 학부생이었다. 그들은 e-mail을 통해 서로간에 더 잘 알게 되었다.


Russia will pay a $15 million (£11 million/€13 million) imposed on them by the International Olympic Committee (IOC) but only if they agree that there will no further punishment taken against the country, Deputy Prime Minister Vitaly Mutko has claimed.
The IOC Executive Board announced the fine as a series of measures after the Schmid Commission ruled Russia was guilty of "systemic manipulation" of the anti-doping systems at events, including the 2014 Winter Olympic Games in Sochi.
The Russian Olympic Committee (ROC) was ordered to reimburse the costs incurred by the IOC during the investigations into the allegations, including setting up another Disciplinary Commission headed by Denis Oswald to deal with individual cases from Sochi 2014, and to help contribute to the establishment of the Independent Testing Authority.
"It is not up to me to accept advice since it is Alexander [Zhukov, President of the ROC], who has to convene a meeting on this issue and take the responsibility," Mutko told Russia's official state news agency TASS.
"I can say only one thing if we pay, then we will pay on the grounds of a concluded deal with the IOC.
"We must not simply transfer the money from our account as we need the reasoning to do so."
Mutko was Russia's Sports Minister during Sochi 2014 and, although the Schmid Commission found no direct link between him and the doping allegations, he was banned from the Olympics for life by the IOC as he head of the Sports Ministry, identified as the biggest culprits in the scandal.
FIFA have claimed that the IOC decision will not affect Mutko's position as head of the Organising Committee for next year's World Cup in Russia or as President of the Russian Football Union.
President Vladimir Putin has promised that he will not order athletes to boycott next year's Winter Olympic Games in Pyeongchang, even though Russia will not be able to take part under their own flag or have the national anthem played if they win a gold medal.
But there continues to be anger in Russia over the IOC punishment.
The State Duma today released a statement condemning the decision.
"The State Duma calls for the cessation of the practice of various kinds of unfounded accusations against Russian athletes and political interference in the sphere of sports," Deputies there said in a statement.
"Sport should be out of politics.
"The ban on representing our country at the competitions limits the rights of the athlete as a person and citizen, which is humiliating for any athlete who is proud to be in favor of his country."
The Deputies in the Duma claimed that those who will suffer most from the decision is "a new generation of Russian athletes".
The statement added: "Such athletes can not and should not be responsible for the previously committed anti-doping rule violations, nor should they feel rogue without national identification, without a hymn, without a flag on the great sports festival-the Olympic Games,"


We've had a crazy weekend here in Sherwood, with wind and rain making it seems like January, not the end of September!
Its put us behind! We need to pull supers off our beehives and get the bees all snugged up for the winter. But its never a good idea to open up the hives and work them in the cold and definitely not when its pouring rain and crazy windy! So here's hoping for a sunny warm day really soon so we can get to work!


text:
\section{Introduction}
\label{sec:introduction}

Deep learning has achieved impressive results in skin lesion analysis, including lesion segmentation, lesion classification, and medical attribute detection. Since 2015, convolutional neural networks (CNNs) are the state of the art for melanoma classification~\cite{data_depth_design}. The standard procedure for training melanoma classification models is to fine tune an ImageNet pre-trained CNN for a melanoma dataset~\cite{transfer_afonso}. However, with the crescent number of CNN architectures, choosing the one to employ is increasingly difficult for researchers.

The ISIC Challenge~\cite{isic_challenge_2016, isic_challenge_2017, isic_challenge_2018}, the largest skin lesion analysis competition, illustrates such increase in the number of available CNN architectures. In the first two editions of the Challenge, the most successful submissions featured mostly ResNet and Inception architectures. In contrast, the most recent, third edition (in 2018) showcased a much wider range of architectures, including also Dual Path Networks, Squeeze-and-Excitation Networks, PNASNet, among others. (We discuss those architectures in Section~\ref{sec:methodology}).

Designing the right CNN architecture for skin lesion analysis (or, in fact, for any image classification task) is far from obvious. Since most tasks tend to reuse/adapt architectures created for ImageNet, the accuracy on that primary task could hint at their accuracy on the target task. Indeed, Kornblith \etal~\cite{do_better_imagenet_transfer_better} evaluated 16 CNN architectures primarily created for ImageNet, and transferred for 12 target tasks, and found a strong correlation between primary and target accuracies.

However, as we will see, their findings must be taken with care when applied for skin lesion analysis. In this work, we reproduce their findings for 9 network architectures over the ISIC Challenge 2017 classification task (melanoma vs. all subtasks) --- but find that the correlations disappear when only the top-performing networks are considered.

Creating ensembles of several models are an effective way of both improving accuracies and stabilizing them~\cite{data_depth_design}. The accuracy of machine learning models tends to fluctuate widely due to uncontrolled nuisance factors, like the choice of the training set, and even random conditions, like the initialization of the networks. In this work, we will also evaluate the performance of ensembles in contrast to single models.

The main contributions of this work are:

\begin{itemize}
  \item We evaluate the factors that affect the choice of a CNN architecture for skin lesion analysis. We evaluate 13 factors over 9 architectures on 5 sets of splits created on the ISIC 2017 classification Challenge dataset.
  \item We evaluate the performance of simple ensemble schemes, contrasting them to single-model performances.
\end{itemize}

All our source code is available on GitHub, to allow the community to reproduce our results, from the training of the networks, until the statistical analyses.

We divided this work as follows: In Section~\ref{sec:related_work}, we review the state of the art of transfer learning in skin lesion classification, and discuss current approaches for choosing CNN architectures. We detail the tools, methods, and CNN architectures used in the experiments in Section~\ref{sec:methodology}. We present the results in Section~\ref{sec:results}, and our conclusions in Section~\ref{sec:discussion}.

\section{Related Work}
\label{sec:related_work}

In this section, we briefly review the literature in transfer learning, and model design, in the context of skin lesion classification. We focus on convolutional neural networks, which are the state of the art on the field~\cite{celebi2019dermoscopy}. For more information, the reader may refer to recent works on deep learning for skin lesion analysis~\cite{michel_towards, celebi2019dermoscopy} and for medical images in general~\cite{litjens2017survey}.

The enormous size of deep learning models contrasts with the scarcity of training data for skin lesion analysis, making transfer learning mandatory for that task~\cite{transfer_afonso, data_depth_design}. Indeed, transfer learning is a commonplace procedure for most computer visions tasks. Compared with training from scratch, knowledge transfer not only increases accuracies but also decreases training times~\cite{rethinking_imagenet_pretraining, do_better_imagenet_transfer_better}.

Most CNN architectures were primarily created for ImageNet~\cite{imagenet} --- a very large dataset, with 1.3 million images and 1000 categories. Due to ImageNet's diversity, those networks learn features that generalize well for other tasks~\cite{what_makes_imagnet_good_for_transfer_learning}. That fact was established by the seminal study of Yosinski \etal~\cite{how_transferable}, which quantified the large effect of transfer learning on accuracies.

More recently, Kornblith \etal~\cite{do_better_imagenet_transfer_better} confirmed those findings with an even stronger result, which established a direct correlation between accuracies on ImageNet, and on the target tasks. They evaluated 16 CNN architectures on 12 target tasks and found strong correlations between Top-1 accuracies on ImageNet, and the accuracies on target tasks. The evaluated architectures comprised 5 variations of Inception, 3 of ResNet, 3 of DenseNet, 3 of MobileNet, and 2 of NASNet. The tasks comprised general-purpose computer vision tasks (CIFAR, Caltech, SUN397, Pascal VOC), and more specialized --- but still aimed at natural photos --- tasks (Birdsnap, Food-101, Describable Textures, Oxford Flowers, Oxford Pets, Stanford Cars). They found very strong correlations for both fine-tuned ($R=0.96$, $\rho=0.97$, p-value $<10^{-8}$) and non-fine-tuned ($R=0.99$, $\rho=0.99$, p-value $<10^{-11}$) models. Curiously, the performance of architectures \textit{without transfer} (initialized at random) also showed some correlation ($R=0.37$, $\rho=0.48$, p-value $=0.03$). That shows that the correlation is due not only to the learned weights but also --- although in a lesser degree --- to the architecture design.

None of the Kornblith \etal's tasks are medical tasks. Studies evaluating the importance of transfer learning for medical applications are few, and for skin lesion analysis even fewer. Menegola \etal ~\cite{transfer_afonso} compared several schemes for transfer learning on melanoma classification and found that fine-tuning a network pre-trained on ImageNet gives the better results when compared with using a network pre-trained on another medical task (diabetic retinopathy). Performing a double transfer (ImageNet $\rightarrow$ retinopathy $\rightarrow$ melanoma) did not improve the results, compared with transferring from ImageNet alone. Using an exhaustive factorial design with 7 factors (network architecture, training dataset, input resolution, training augmentation, training length, test augmentation, and transfer learning) over 5 test datasets, for a total of 1280 experiments, Valle \etal ~\cite{data_depth_design} showed that the use of transfer learning is, by far, the most critical factor. In a factorial Analysis of Variance, it explains 14.7\% of the absolute performance variation and 62.8\% of the relative variation (excluding residuals and the choice of test dataset), with high significance (p-value $<0.001$).

In any event, transfer learning and fine-tuning are heavily used for skin lesion classification. All top-ranked submissions for ISIC Challenges 2016~\cite{isic_2016_first_place}, 2017~\cite{isic2017_titans_submission, isic2017_participant_second, isic2017_participant_third, isic2017_participant_first_kerat} and 2018~\cite{isic2018_participants_first_with_data, isic2018_participants_second_with_data, isic2018_participants_first_no_data, isic2018_participants_second_no_data, isic2018_titans_submission} used CNNs pre-trained on ImageNet.

While there is a consensus on the use of transfer learning for skin lesion models, when choosing the architecture no choice is universal. On the contrary, the classification task of the ISIC Challenge shows a progressive \textit{diversification} of architectures. In 2017, the top four submissions used two networks: ResNet~\cite{isic2017_titans_submission, isic2017_participant_second, isic2017_participant_third, isic2017_participant_first_kerat}, and Inception-v4~\cite{isic2017_titans_submission}, while the latest challenge~\cite{isic_challenge_2018}, in 2018, showcased a wider range of choices among the top performers --- not only ResNets and Inceptions, but also DenseNet~\cite{isic2018_titans_submission}, Dual Path Networks~\cite{dual_path_networks}, InceptionResNet~\cite{inceptionv4}, PNASNet~\cite{pnasnet}, SENet~\cite{senet}, among others --- usually in ensembles of multiple architectures~\cite{isic2017_titans_submission, isic2018_participants_first_with_data, isic2018_participants_second_with_data, isic2018_participants_first_no_data, isic2018_titans_submission}.

The best architectures for skin lesion classification remain, thus, an open issue. The Challenge results do not allow analyzing a single factor from a multitude of choices among participants. The study of Valle \etal ~\cite{data_depth_design}, although exhaustive for 9 factors, evaluates only two levels for each factor, picking ResNet-101 and Inception-v4 for architectures. No other study, as far as we know, attempts to answer the question systematically.

In this work, we focus on that issue, by applying the design of Kornblith \etal~\cite{do_better_imagenet_transfer_better} to the task of melanoma classification. We evaluate the performance of 9 networks (listed in Table~\ref{tab:architectures}) pre-trained on ImageNet, and fine-tuned for melanoma classification, on the dataset of the ISIC 2017 challenge.

In comparison to Kornblith \etal's, our study reduces the scope of the tasks and enlarges the scope of the correlations attempted. On the one hand, while they evaluate 12 general and specialized natural-photo tasks, we focus only on melanoma classification. On the other hand, while they correlate only the Top-1 ImageNet accuracy to the target accuracy, we correlate several network attributes and source and target metrics. We aim to obtain hints on how to select the best architectures for melanoma classification.

Since ensembles of models have special importance in the literature of melanoma classification, we also evaluate how simple ensembles of the evaluated models perform in comparison to the models alone.

\section{Methodology}
\label{sec:methodology}

\subsection{Data}

All data comes from the ISIC Challenge 2017 dataset~\cite{isic_challenge_2017}. We do not employ the original training (2000 images), validation (150) and test (600) splits. Instead, we combined all 2750 images and produced five random combinations of train (1750), validation (500), and test (500) splits --- aiming at an approximate 60--20--20\% partition. We chose those proportions because the original validation set was too small to allow making decisions on the deep learning hyper-parameters with confidence, and we made five completely random combinations to allow estimating the variability due to the choice of the splits. The exact images used in our splits are available in our code repository (see next section).

Although the ISIC Challenge 2017 provided annotations for three classes (nevus, seborrheic keratosis, and melanoma), and had three subtasks, we focus only on the subtask of melanoma vs. all.

\subsection{Tools}

We evaluated 9 different architectures (Table~\ref{tab:architectures}), whose PyTorch implementations and pre-trained ImageNet snapshots we obtained from other authors. The choice of publicly available snapshots reflects current practice, since retraining the architectures from scratch on ImageNet is very time-consuming. Kornblith \etal report major performance increases for architectures carefully trained from scratch on ImageNet, for transfer learning \textit{without} fine-tuning. For transfer with fine-tuning (which we evaluate in this paper), they found little difference between public snapshots and models trained from scratch.

Table~\ref{tab:architectures} shows the architectures we chose. Except for MobileNetV2, we chose the architectures for their performance on both ImageNet and the ISIC Challenge. We kept MobileNetV2, present in the original study by Kornblith \etal, since we found interesting to include an architecture aimed at embedded and mobile hardware.

The only modification on the architectures was changing the last layer from the 1000 ImageNet classes to a binary classification layer.

We fine-tuned each network with stochastic gradient descent (SGD) with a starting learning rate of $1\mathrm{e}{-3}$ and momentum factor of $0.9$. All layers were left free to evolve. Whenever the validation loss failed to improve for 8 consecutive epochs, we divided the learning rate by 10. We stopped the tuning if the validation AUC (area under the ROC curve) failed to improve for 16 epochs. To avoid accommodation on the training data order, we shuffled them before each epoch.

We followed the best data augmentation settings proposed in~\cite{dataaug_for_skin}: random horizontal/vertical flips; random cropping; rotation up to $90\degree$; shear up to $20\degree$; area scaling from $0.8$ to $1.2$; and color (saturation, contrast,
brightness, and hue) jittering. We also used augmentation on test with 64 copies, and on validation with 16 copies (average-pooling the probability vectors of the copies as the final result). We resized each image according to the input expected by each architecture ($224\times224$ for DenseNet, ResNet, DualPathNet, SENet, and MobileNetV2; $299\times299$ for Inception-v4, InceptionResNet-v2, and Xception; and $331\times331$ for PNASNet). The inputs were also normalized per-channel using the z-score computed with the training dataset statistics.

We used an NVIDIA Tesla P100 with 12~GiB to fine-tune all models. We considered the memory GPU as a constraint to our experiments, and we picked, for each method the largest batch size (in multiples of 8) that the model could fit (Table~\ref{tab:architectures}). Although in a theoretical setting we could have compared all models with the same batch size, we considered our criterion more useful for practical purposes, since occupying the GPU memory as much as possible is the usual procedure. Considering that practical setting, our criterion is ``fair'' in the sense that it allows considering a compromise between larger models vs. larger batches.

All the source code used in this paper, from model tuning until statistical analyses, is available in our public repository\footnote{\url{https://github.com/learningtitans/cvpr-skin-solo-ensemble}}. Our code is easily adaptable to allow new architectures.

\newcolumntype{Z}{>{\raggedright\arraybackslash}p{0.23\linewidth}}
\newcolumntype{K}{>{\justifying\noindent\arraybackslash}p{0.41\linewidth}}
\newcolumntype{R}{>{\raggedleft\arraybackslash}p{0.052\linewidth}}
\newcolumntype{B}{>{\raggedleft\arraybackslash}p{0.06\linewidth}}

\begin{table*}[!htbp]
\centering
\begin{tabular}{ZRRBRK}
\toprule
\textbf{Architecture} & \textbf{Acc@1} & \textbf{Acc@5} & \textbf{Params (M)} & \textbf{Batch Size} & \textbf{Summary} \\ \midrule
\textbf{DenseNet}~\cite{densenet}~\textsuperscript{A} & 77.6 & 93.8 & 28.7 & 40 & Composed of dense blocks, which concatenate the output feature map of each layer to all subsequent layers. \\
\textbf{Dual Path Nets}~\cite{dual_path_networks}~\textsuperscript{B} & 79.8 & 94.7 & 79.3 & 24 & Combines ResNet's residual paths for feature re-usage and DenseNet's dense connections for new features exploration. \\
\textbf{Inception-v4}~\cite{inceptionv4}~\textsuperscript{B} & 80.2 & 95.2 & 55.8 & 64 & Composed of Inception modules, which have parallel convolutional layers that learn different cross-channel and spatial correlations. \\
\textbf{Inception-ResNet-v2}~\cite{inceptionv4}~\textsuperscript{B} & 80.1 & 94.9 & 42.7 & 32 & Similar to Inception-v4, but with residual connections. \\
\textbf{MobileNetV2}~\cite{mobilenetv2}~\textsuperscript{C} & 71.8 & 91.0 & 3.5 & 128 & Uses depth-wise separable convolutions to produce an efficient network, suitable for mobile devices. \\
\textbf{PNASNet}~\cite{pnasnet}~\textsuperscript{B} & 82.7 & 96.0 & 86.1 & 8 & Designed with modules discovered through Neural Architecture Search (NAS) (current best accuracy on ImageNet). \\
\textbf{ResNet}~\cite{resnet}~\textsuperscript{A} & 78.4 & 94.1 & 60.2 & 56 & Uses residual connections to improve information flow, allowing networks with more than 100 layers. \\
\textbf{SENet}~\cite{senet}~\textsuperscript{B} & 81.3 & 95.5 & 115.1 & 24 & Composed of Squeeze-and-Excitation (SE) blocks, which capture channel-wise dependencies for convolutional features maps. \\
\textbf{Xception}~\cite{DBLP:conf/cvpr/Chollet17}~\textsuperscript{B} & 78.9 & 94.3 & 22.9 & 40 & Extrapolates Inception modules to depth-wise separable convolutions, resulting in more efficient parameter use. \\ \bottomrule
\end{tabular}
\caption{CNN architectures used in the experiment. Acc@1 and Acc@5: performances on ImageNet. Params: number of trainable parameters (in millions). Models and checkpoints sources (superscripts on model names): A) \textit{github.com/pytorch/vision}; B) \textit{github.com/Cadene/pretrained-models.pytorch}; C) \textit{github.com/tonylins/pytorch-mobilenet-v2}.}
\label{tab:architectures}
\end{table*}

\subsection{Experimental Design}
\label{sec:exp_design}

For the main experiment, evaluating the effects on the choice of the architecture, we make 3 repeated experiments (tuning and measurements) for each of the 9 architectures, on each of the 5 sets of splits. The 3 repeated experiments allow evaluating the effect of random choices: initialization of the last layer, dropouts, data augmentation, shuffling of data on epochs, etc. The main experiment, thus, comprises $3\times9\times5=135$ measurements of several metrics: AUC, accuracy, sensitivity, and specificity for both the validation set (at the epoch chosen by the early stopping procedure) and the test set. We also measure the loss at the validation at the epoch chosen.

For the analysis of the main experiment, we employ a correlogram (Figure~\ref{fig:correlogram}) of the metrics above, adding 7 attributes of the architectures: Top-1 accuracy on ImageNet, time of publication, number of parameters, and number of the epoch picked by the early stopping. In order to make the correlations comparable across the 5 sets of splits, we perform an adjustment similar to the one used in a repeated measures/within subjects Analysis of Variance: we subtract from each metric on a given experiment its average across all experiments in the same set of splits, and add back its average across all experiments. We consider the correlations significant when their confidence intervals do not contain zero. For the confidence level, we employ a Bonferroni-adjusted  $\alpha=0.05/78$, where 78 is the number of pairs of variables in our correlogram.

For the ensembling experiments, we employ the base models above. For each of the 5 sets of splits, there are 27 single models. We create the ensembles by ordering those 27 models, choosing a number $n$ between 1 and 27, and combining the first $n$\textsuperscript{th} base models into an ensemble. To simplify the evaluation, we use a very simple, but effective~\cite{data_depth_design} strategy of average-pooling the output prediction probabilities for the ensemble.

We contrast two strategies: ordering the base models by their AUC on the validation set and ordering them at random. We replicate the latter 10 times, to evaluate the variability. In all cases, the measurement performed is the AUC on the test set. The experiment aims to determine if the AUC on the validation set is useful to choose the models for the ensemble.

In order to evaluate the results, we employ two plots: in one we contrast the ensembles ordered by the validation set vs. the ensembles ordered at random separately for each of the five sets of splits. In the other, we gather all five splits in a single series for each type of ensemble (validation vs. random) using a repeated measures/within subjects procedure like the one explained above.

\begin{figure*}[t]
\centering
\vspace{-0.6cm}
\begin{subfigure}[t]{.63\linewidth}
  \centering
  \includegraphics[width=1.\linewidth]{figures/networks_correlogram_within_splits.pdf}
\end{subfigure}
\\
\begin{subfigure}[t]{.63\linewidth}
  \vspace{-1.6cm}
  \centering
  \includegraphics[width=1.\linewidth]{figures/networks_correlogram_within_splits_without_mobilenet.pdf}
\end{subfigure}
\vspace{-0.6cm}
\caption{Correlograms of the network attributes and outcome variables. Top correlogram: with MobileNetV2; bottom correlogram: without it. Acc@1: Top-1 accuracy on ImageNet. On each correlogram, the upper matrix has the scatter plots and the lower matrix has the Spearman's $\rho$ correlations (positive in black, negative in red). The area of the circles also indicates the magnitude of the correlations, the dashed circles indicating the confidence interval for $\alpha=0.05/78$ (78 = Bonferroni correction). For intervals containing zero, we omitted the circles, indicating non-significant correlations.}
\label{fig:correlogram}
\end{figure*}

\begin{figure*}
\centering
\begin{subfigure}[t]{.82\linewidth}
  \centering
  \includegraphics[width=1.\linewidth]{figures/ensembles_plot.pdf}
\end{subfigure}
\\
\begin{subfigure}[t]{.82\linewidth}
  \centering
  \includegraphics[width=1.\linewidth]{figures/ensembles_plot_within_splits.pdf}
\end{subfigure}
\caption{Experiments with ensembles of models. Solid lines: models with best validation AUC chosen first. Dashed lines: models chosen at random (Section~\ref{sec:exp_design}). Thick lines: averages, Shadows: standard deviations, Thin lines: individual runs. \textbf{Top}: Experiments separated per split (colors). \textbf{Bottom}: Split differences marginalized, and experiments grouped by type of ensemble. There is a slight advantage in using the validation set to choose the models, but choosing at random also provides good results.}
\label{fig:ensembles}
\end{figure*}

\section{Results}
\label{sec:results}

The correlation analyses are in Figure~\ref{fig:correlogram}, the results on the topmost correlogram appear to partially confirm the results of Kornblith \etal. Indeed, the first column show positive significant Spearman's correlations ($\rho$) between the Top-1 ImageNet accuracy with almost all the target measure metrics: accuracy, AUC, sensitivity, and specificity. However, a more careful inspection of the data --- on the scatter plots of the first line --- raises questions about that conclusion. The positive correlations seem to be due to a single group of samples having detached from the rest of the data. In addition, while Kornblith \etal. observed very high correlations ($\rho>0.95$), the ones we found, although significant, are modest, to say the least ($0.33<\rho<0.50$). The number of parameters in the network also show significant modest correlations with most metrics ($0.30<\rho<0.44$).

The correlogram on the bottom of Figure~\ref{fig:correlogram} further contradicts the findings of Kornblith \etal. By removing MobileNetV2 from the analysis, all significant correlations between ImageNet accuracy and target metrics disappear. Not even a general tendency appears: half the remaining (non-significant) correlations are positive, and the other half negative. The correlations with the number of parameters also become non-significant, although a general tendency still remains: the non-significant correlations remain mostly positive.

On both correlograms, we observe a general positive correlation between the metrics, clearer on the top correlogram (with MobileNetV2). The exception is specificity, which not only shows the expected anti-correlation with sensitivity, but also tends to correlate negatively with most other metrics. Those tendencies had been already observed by Valle \etal~\cite{data_depth_design} on the evaluation of two architectures.

The most useful significant result in both correlograms is the positive correlation between the metrics on the validation set and on the test set --- especially when contrasted with the non-significant or much smaller correlations between the validation loss and the metrics. That suggests that the validation loss might be a poor proxy for \textit{any} of the metrics, and that using the actual metric to make decisions (e.g., on early stopping) might be a better plan.

Two results on the correlograms reinforce the importance of ensembles for skin lesion classification: first, the impossibility of establishing any definitive criterion to select an architecture as a definitive choice. Second, the large amount of variability due to uncontrollable sources (\ie choice of the folds and random nuisances).

The results of the experiments on ensembles are on Figure~\ref{fig:ensembles}. In the topmost plot, the results are separated for each set of splits (each represented by a different color) --- the most influential factor in determining the results. For each set, however, we can see how choosing models ordered by the validation split tends to produce better ensembles.

The bottom plot marginalizes over the splits for both types of ensembles, and shows how the ensembles sorted by validation have a slightly better mean (thick lines) and smaller variability (shaded area). It is remarkable, however, how even ensembles of enough models chosen at random have surprisingly good performance.

In both plots, we can see how variability decreases as the ensembles incorporate more models. Those estimations, however, must be interpreted carefully: the variability decreases because there is a limited number of available base models, and thus the ensembles necessarily become more alike as their number of base models increase --- at 27 base models, all ensembles become the same, and any variability disappears.

\section{Discussion}
\label{sec:discussion}

We were disappointed our results failed to confirm those found by Kornblith \etal\ --- had we found the same results as them, we could employ the accuracy on ImageNet as a safe proxy to choose architectures for skin lesion analysis. In our results, however, not only no network characteristic (e.g., number of parameters) correlates well with the target metrics, but also most networks appear impossible to distinguish from one another in an statistical test. Uncontrollable factors such as the choice of splits on the dataset, and even random nuisances appear more influential than the choice of architectures. That analysis however, is only true for a selection of very high-performance architectures. When we add a shallower, less complex architecture (MobileNetV2) to the lot, it appears different than all others --- to the point that just by creating a set of very different measurements, it can make several of the correlations significant. Our results certainly do not imply that one can select an architecture from 2013, and expect 2019-level performances.

It is not obvious why the findings of Kornblith \etal do not reappear in our study. On one hand, our study has an important limitation: the size of the dataset. All datasets employed by them were larger than ours (although one of them had a training set even smaller than ours). In a follow-up study, we would like to do our correlations over several datasets, including the full ISIC Archive. On the other hand, \textit{their} study has also an important limitation: they did not evaluate several sets of splits for each dataset, neither multiple trainings for each network. We found those uncontrolled factors to be the main sources of variability, largely reducing the correlations. Finally, an important distinction between both studies, is that they perform extensive tuning of the training hyper-parameters of their models, while we adopt a standard approach considered ``best practice'' for most models. We think those choices do not necessarily reflect a limitation of either study, but different aims. Instead of extensive tuning to a particular skin lesion dataset, we opt for several attempts, to reflect the variability expected on real-world scenarios. They, however, are evaluating ``classical'' computer vision tasks, where those extensive tunings are expected to reflect models present in existing literature.

Those results reinforce the importance of ensembles of diverse architecture as the preferred mechanism to obtain good models for skin lesion analysis.  Our results show that for small ensembles it is very useful to employ the validation set to select the best base models, but that for large ensembles one can possibly get away simply choosing the models at random.

Although the aim of this paper was not to maximize any of the measured metrics, the plots on both Figures~\ref{fig:correlogram} and~\ref{fig:ensembles} help as sanity checks, to verify that our models' performances are not unrealistically low compared to existing art. The melanoma vs. all AUCs of the single models evaluated was between 84 and 91\% (86 and 91\% without MobileNetV2). The first place on the ISIC 2017 Challenge was 87.4\% --- almost exactly the average value we found for our ensembles.

\section*{Acknowledgments}

S. Avila is partially funded by Google LARA 2018. E. Valle is partially funded by a CNPq PQ-2 grant (311905/2017-0). This work was funded by grants from CNPq (424958/2016-3), FAPESP (2017/16246-0) and FAEPEX (3125/17). The RECOD Lab receives addition funds from FAPESP, CNPq, and CAPES. We gratefully acknowledge NVIDIA for the donation of GPU hardware.

{\small
\bibliographystyle{abbrv}

redpajama_set_name: RedPajamaArXiv


text:
Marco Silva charged with improper conduct after dismissal in Bournemouth clash
Fulham manager Marco Silva has been charged by the Football Association over his alleged improper behaviour towards the end of Saturday's 1-1 Sky Bet Championship draw at Bournemouth.
The promoted Cottagers took the lead through their 41-goal striker Aleksandar Mitrovic before the Cherries were awarded a penalty deep into stoppage time.
Silva was dismissed in the aftermath of the spot-kick decision before Dominic Solanke scored from the spot.
The Portuguese boss said in his post-match interview: "The decision to award the penalty was a mistake from the referee. That made a big impact in the game.
"We wanted to come here to be champions and to get the three points. We did not deserve that mistake.
"For me it was tough to get sent off in that moment."
An FA Spokesperson statement read: "Fulham's Marco Silva has been charged with a breach of FA Rule E3 following their Championship game against Bournemouth on Saturday.
"It is alleged that the manager's behaviour during the 95th minute of that game was improper, and he has until Friday to provide a response."
Katie Price opts for trial after denying breach of restraining order
Ronnie OSullivan dismisses superstar tag after cruising into last four