>

It's not sus at all, but the artisan handcrafted headline will make you think that.
Not surprised. That’s why you use a test environment before implementation. On something as big as the FAA probably roll it out in parts too. >

That's not fair. They had a test system.
They just didn't have a separate production system. >

In every company I've ever worked for I always ask if we have a clone of production for testing.
Only once have I been told "yes".
Every other time, I'm told that the QA environment is "close enough". >

Until it isn't.  But why spend money doing something useless until it costs you millions of dollars and/or customers? >

Why wouldn’t you just make a new copy of the production environment and overwrite the QA environment with it so you have a test system that is up to date to test your code? >

The biggest hurdle isn't the code, it's the volume of users that exist in Prod vs in sub-prod environments.
And just straight copying prod data down isn't a great idea, because it contains PII. >

I have worked in QA for over a decade and I fully support this answer. With more users/volume comes more edge case usage. >

Also 3rd party integrations might not work in QA environment. Ideally they should, but sometimes that requires extra setup or work with the vendors. >

And licensing is usually the issue here.
3rd party contractors aren't going to take a missed opportunity to charge for the licensing for the test environment at the same scale as the production environment. >

Yet another argument for permanent full time employees. 
If it’s news worth mentioning their terms of employment, then that’s exactly the issue and problem. >

The FAA utilizes multiple means of communication to disseminate information to airlines. A major one is the FAA OIS page which publishes programs and other significant events occurring in the NAS. Airlines have access to a better version of this with more detailed information as well as datalinks to systems that monitor demand, loading, and throughput of airspace and airports.
ATC does national teleconferences every 2 hours and will sometimes activate that same infrastructure in times of emergency. There was a teleconference with all carriers during the night the NOTAM system went down. >

The system is from 1993. The same year the first internet browser became publicly available, one year before the original Playstation, and three years before HoTMaiL (get it? Do you get it?).
How old were the contract employees? Old enough to even remember 1993? Were they even born yet?
Even a dev who was experienced in 1993 has spent the last 30 years learning and adapting to the evolutions in technology. I challenge anybody to go to work and do their jobs with Office 4.0, or operate their computer with MS-DOS. Even if you're old enough to have been expert at it then, you certainly would struggle with it now. 
The issue isn't the contract employees. The issue is that our government has been trying to repair our critical infrastructure with 10,000 band-aids, and if even one of them falls off then our economy starts bleeding to death.
How many of our other critical systems are literal antiques that could break at any moment? Does anybody have the skills to repair them anymore, or are those people all in nursing homes by now?
Do I even want know the answer? >

I’m not sure that this is a huge deal like you think it is, NASA uses incredibly old hardware designs because critical infrastructure needs to be tried and tested and reliable. Granted they have the difficulties of increased radiation from space to deal with, but still even on the james webb telescope launched semi recently a processor released in 2001 was used. Granted I know almost nothing about flight infrastructure/systems >

Are you sure it's not an issue? Every single plane in the country was grounded because of this. Didn't SW just have a huge debacle because of outdated systems? And didn't the unemployment systems crash in 2019 and no one could fix them because nothing is programmed in COBOL anymore? Isn't a huge part of the increased funding for the IRS to update their systems because they're also outdated and can't handle the load?
Comparing a one-off science experiment that has a built-in life expectancy of 10 years or less to a critical infrastructure system that requires continuous functionality to enable our modern way of life is really apples to oranges here. >

To be fair all the issues you mentioned rarely happen and the systems running those programs are ancient. So they are highly reliable. >

Sure, that's a fair point. But these systems aren't going to last forever, they are going to need an upgrade, and probably should have been upgraded already, maybe even years ago by now. The can keeps getting kicked down the road, and that's only going to make things worse. >

maybe hire people full time rather than a churn of contractors who always need to ramp up when onboarding? >

I can tell you who's not getting contracted again. >

Don't be sure. Part of contracting is taking the heat for the stupidity you encounter. >

The other part is donating to the right politicians so your company is legally the only one they can use >

What do you want to bet the "corrupted file" was a config, recently edited with vi, that had an accidental / in it?  Possibly as the first character? >

Do they not have a UAT environment to test changes? >

UAT environments only help if they test for that issue. >

Reminds me of the “contractors” who’s code spoiled an entire line of Boeing aircraft >

Not to mention the passengers that were inconvenienced by a sudden and catastrophic deceleration .  /s >

I think the world banking system mostly runs on COBAL still. You can’t change languages every decade or nothing would work. This sounds like poor management at the top and lack of funding. >

Back in 2020, NJ was looking for COBOL programmers because the unemployment system still ran on it. I think they've updated it since but I'm not sure. >

I was a contractor working in the air transportation industry at SITA. Positions like this should not be held by contractors, it should be a full time employee, one of which the FAA invests in. Contractors do just enough to continue the engagement, knowing that their future development doesn't matter to anyone else. >

I don’t usually test my code, but when I do I test it in PRODUCTION. >

Yesterday they were reporting it was an accidental file deletion... >

I wouldn't be surprised if it doesn't come out that this was a Russian attack. They've been targeting other NATO infrastructure with their spy games.
Not saying it was, just that it wouldn't surprise me. >

Ate you fancying that it was a bear? >

Penny wise pound foolish. >

Shifting blame. The company is responsible for everything, they should use better risk and quality control. >

Also I bet they layed off devs that understand how everything in the outdated , probably spaghetti,  code base worked and tried to replace then with contractors. That's how the private sector works anyway. >

I work in software. It's terrifyingly easy to make these kinds of mistakes on these kinds of systems. Lack of test environments due to budget, very simple systems where a single misplaced semi-colon might bring down a system because it's not robust enough to detect errors and work around them. I've seen things as small as capitalization on a server name prevent large systems from operating properly. >

Our infrastructure is so outdated and barely held together, and the only time it gets upgraded is when we, the taxpayer, foot the bill for it. 
This goes for everything. Airlines, ISPs, Hospitals, Utility companies,  etc. 
This shit is just packaged a different way each time to give the impression of progress, but at the end of the day the big players in the game still make out like bandits. They build the worst fucking infrastructure and then parachute out of the plane to let the next generation deal with it. 
Its a universal "I got mine" from the upper class to the struggling middle and lower class. 
Its like ordering the most expensive shit at a restaurant and then walking out and sticking us with the bill. >

Our government used to hire the best and brightest, now they hire contractors that they are told are the best and brightest.
What to see when America started to fall? look to Regan. >


Our government used to hire the best and brightest, now they hire contractors that they are told are the best and brightest.

You should look at how much the government pays their STEM workforce. I looked into an electrical engineering job with the Air Force civilian service out of school, but they wanted to pay me $30,000--frankly that's insulting. I can make more than that at McDonalds. I accepted a private sector offer for 2.5 times that, that has bonuses, profit sharing, and stock options. >

That’s all by design and that’s the problem. >

It’d be peak govt to pay that contractor tons of money to fix the system that they themselves fucked. 
Source: govt employee >

It's amazing how that works. A contractor is hired to do a job, fucks it up and gets paid extra to fix it.
Source: former govt contractor who didn't fuck up - shakes head. >

Mainframe system? COBOL and Assembly code? >

Fortran too? >

What a joke, I had at least a few idiots arguing with me in the comments when this happened that the FAA was actually competent and that the billions in profits/handouts had, in fact, actually been reinvested properly. We see now that is not the case.
Friendly reminder - the last 3 heads of the FAA have all been former VP's of operations at airllines/airline industry groups >

If one corrupted file get bring down your whole system including backups, you have bigger problems than contractors making mistakes. >

They could be related, but is it possible that NAV Canada shut theirs down out of an abundance of caution and said it was a "technical " issue until they confirmed with the US that it wasn't nefarious? >

This is why you dont just blindly copy code from Stack Overflow and put it into production . >

Please tell my coworkers this. They won't listen to me. I even found code that had comments from stack overflow facepalm (and the code was shit and shouldn't have been the accepted answer!) >

Some days when I am reading the news I ask what exactly the military is protecting us from. The classified documents with Trump and now Biden, the power substation attacks and now this. Perhaps less focus and money on killing toys like stealth jets and more on actual security threats. I feel like the nation has Uvalde PD as security. Not a good feeling. >

I told him not to download it from the torrent. Damn you. >

so the ‘FAA FlytZZZ’ app isnt legit? >

Do software contractors not know about sandbox environments? Where’s the usability testing? QA? Unit Tests? >

Lol i bet you they just wrote some simple unit tests like checking if the div renders instead of mock objects to test out the logic itself. >


Tens of thousands of travelers were left stranded Wednesday after the FAA sent out a tweet at 7:20 a.m. ordering the airlines to pause all domestic departures until 9 a.m. ET “to allow the agency to validate the integrity of flight and safety information” as it worked to restore the NOTAM system.

Ummm... The FAA communicates emergencies through twitter? TIL >

The sentence implies a relationship between the tweet and the travelers left stranded, which is not true. What is true is that the FAA sent out a tweet, and some time after that tweet travelers were left stranded due to (I presume) airlines canceling certain flights because of crews timing out or other issues. 
The way the FAA actually disseminated the operational information was a message sent to air traffic control facilities telling them not to allow aircraft to depart. That message was not sent over Twitter. >

Thank you for the reply. >

Well... look at that.... 
​
seems like someone needed in-house developers and that changing a team and adding new people to a mess only costs more.... probably a lot of Brook's law.. >

They needed to update a system 10 years ago. They didn't. >

Well this isn’t sus at all