Feds at the Edge

Digital transformation is a phrase bandied about everywhere these days. Its application is everywhere from ordering coffee online to the Pentagon.  Just as a point of comparison, the DoD has 27 million people with over $3 trillion in assets and over 17 business systems.  The amount of complexity is overwhelming for any project involving change.

Today’s interview brings together four experts in improving the existing Enterprise Resource Management (ERP) system in the federal government. The discussion gives the listener insights into the areas of strategy, automation, and process improvement.

The DoD is comprised of individuals with strong leadership characteristics. It is possible that each leader promotes their agenda at the cost of the others, and, at the expense of the overall agency objective. During the interview today, Greg Little, DoD, reinforces the new concept of starting with outcomes. 

Once this is done, duplicative systems can be eliminated, and systems can be designed to reduce costs for the entire portfolio. The phrase that sums it up is, “have the business strategy drive the technology strategy.”

One aspect of accomplishing the digital transformation of an immense organization like the DoD is automation.

Greg Little provides more details on the complexity of IT at the DoD. There are over 25 accounting systems with more than 250 feeder systems.  Conducting an audit with this level of complexity would never be possible with a few accountants and some spreadsheets.  Automation is the only way to handle this number of systems, without it, the audit would not be sustainable.

The DoD is currently using Robotic Process Automation, and have 50 digital assistants.  Of course, the goal is compliance with financial regulations, but automation allows leaders to use data to drive decisions, tie resources to performance, and financial stewardship.

Part and parcel of these large systems are legacy ERP.  Jonathan Moak, Salesforce suggests that older systems must be incorporated into the move to the cloud. However, if can’t merely be a “lift and shift,” it must make finance part of the complete business management system. 

Enhanced flexibility and interoperability will allow leaders to gain a better understanding of the impact of financial management on the entire agency.

IBM and the Ponemon Institute have reported that the average breach cost in 2022 was $4.24 million. If that is used as a rough estimate, data breaches in state and local governments can be very expensive. There is a lot at risk and budgets are tight. State and local governments really have to look at leveraging what funds they have.

Today’s discussion provides recommendations for sources of information on hardening systems, coming up with action plans, and the role of insurance.

There is no lack of help if you are seeking guidance when it comes to making your system secure. Guides from CISA and NIST give specific information.  Most suggest starting with an accurate evaluation of what is on your system.  There may be situations where people sign up for services with a credit card without informing system managers. System surveys are difficult when one has to look for shadow IT.  

Action plans normally start with ways to respond to an incident.  One weakness in a backup playbook is the time it takes to restore one system vs. ten systems. System managers may have to get ideas on unexpected circumstances.

Best practice is to harden your system and have an action play. The unintended benefit of documenting your security is qualifying for cyber insurance.  Risk assessment can vary in size of organization.  Insurers try to limit exposure – excluding certain events. One certain bet is that it will become more and more expensive to get cyber insurance.

You may not realize that a cyber insurance package can be an 11 page application   combination of entire system – every environment will have a different footprint. Tony Lauro Akamai mentions that an insurance plan must never be considered to be a substitute for a hardened system. 

Interest in machine language and artificial intelligence has been growing and growing since around 2015. It was a perfect storm where storage prices decreased, and virtualization became standard.  Like most maturing industries, challenges appeared. It was found that the conclusions provided by artificial intelligence were dependent on the quality of the data it collected. This finding became so common that in 2020 the federal government responded with Executive Order 13960 promoting the use of trustworthy intelligence in the federal government.

This is an interview with a focus on applying that guidance across many federal areas, including cybersecurity.

Technology leaders from the National Science Foundation have formulated the National Artificial Intelligence Research Resource Task Force (NAIRR) https://www.nsf.gov/cise/national-ai.jsp  to give guidance on using data more effectively. Recommendations include the categories of security privacy, civil rights, and sustaining the resource. The goal was to help get control of the unwieldy and expanding IT environments.

Wayne LeRiche, Palo Alto Networks, gives an example of applied AI with some of the security concerns relating to Domain Name Servers (DNS).  Attacking a DNS server is a classic approach malicious actors have used for years.  The traditional method of defense is rule-based.  If “x” occurred, then react with “Y.”

Attackers can ratchet up the attack with something unknown that the rules-based system can’t handle it.  AI can handle drastic increases in speed that a rules-based system can’t. Further, artificial intelligence can understand unknown methods of denying service to the DNS.

Rather than focusing on one aspect of prevention, experts on the panel suggest that Rule-Based and Machine-Learning DNS security services be used in parallel.

Finally, Tony Walker, NetScout, presents a scenario where one part of the team is proficient in network management while another part of the team has a sophisticated knowledge of data management. The solution that is provided by AI must be tempered with careful consideration of data sources and the ability to use that knowledge in an environment full of people with different skill sets.

Both federal and civil organizations are challenged with coming up with creative ways to keep and attract a modern workforce. During this discussion, federal leaders share innovations on retaining their workforce and replacing retiring employees.   

Paul Pietsch, Partnership for Public Service brings up some shocking statistics.  In one agency, the Gen-Z makes up only 1.6% of the workforce and, in the next two years, 33% of the federal workforce in that agency will hit retirement age. This second figure is sometimes referred to as the retirement tsunami.

The experts on the panel all admit that they are presented with a challenge of retaining the younger workforce, and, at the same time, reaching out to new candidates.  The only way to reach this goal is with forward thinking approaches. 

When it comes to keeping Gen X and Gen Z on board, the federal leaders discussed topics like rotating assignments, appealing to public service, assisting with student loans, including retention bonuses, flexible scheduling, and mentoring.

Post COVID, all agencies understand the attraction of remote working and having a flexible schedule.  One creative idea that is used at the Department of Education is rotating assignments.  A manager can offer an inducement for a person to stay with the agency by offering them the ability to learn new skill sets in other areas.

Another innovation suggested by Jacqueline Clay, Department of Education, is to form cohorts of people when they get hired. You can select them for specialized training.  Because they will get to know one another, they can share knowledge in an informal manner.

Mentorship programs have been started in some agencies, normally they last one year.  In a twist, some are experimenting with reverse mentorship: where a younger person can share knowledge of topics like Slack with professionals who may have expertise in other areas.

Some agencies offer financial incentives to keep team members and will compensate for a referral that results in a hire.

Today’s young people can have college debt, some studies show the average student debt is $18K.  Some agencies are offering up to $60K in student loan forgiveness.

If your target employees are participating in social media, then a wise leader should know how to communicate in that media.  Joseph Abbott, from the USDA, remarked they created a new vocabulary where agency concepts could be understood in a world of emojis and memes.

Federal mandates include Environment, Social, and Governance (ESG) policies. Like many federal initiatives they provide broad guidance and enacting these lofty goals can be difficult.

Today’s interview looks at leaders from the federal government, state, and local who share ways they have been able to include ESG goals in their organizations

Two of the people in the podcast are from California. Traditionally, California has been a leader in these efforts for decades.  Andrew Collins, from the San Francisco Employee’s Retirement System, really sums up the challenge.  Because he has a fiduciary responsibility to the people who retire, he tries to meet ESG goals, but must include risk management along with seeking opportunities for ESG characteristics.

The world of finance and investment is full of metrics.  One challenge the discussion participants discussed is how to measure whether investments include climate change or even social equity goals. Brian Rice from the California State Teachers’ Retirement Funds lets listeners know about the 25 risk factors included in the evaluation his group uses.

Most people know that the National Science Foundation provides grants for a wide variety of needs. In response to a recent Executive Order indicating that ESG goals must be included in grant approval, the NSF has taken a slightly different tack than the state and local groups.

The NSF is looking at geographic targets for development well.  For example, they are trying to create geographically diverse communities for investment in the NSF project.  The idea is that no equity is accomplished if only one area has all the innovation.  They layer ESG goals on top of this regional approach.

One innovation from the NSF is to create a repository of information about technology projects. For example, if a person is seeking funding for a specific biotechnology project, they can easily see where similar innovators are considering launching companies.  This eliminated duplicative efforts and allowed for collaboration beyond one’s locale.

 According to a recent survey from KMPG, ESG considerations about ESG are reflected in these recent Executive Orders. Results indicate 70% government should take a role in solving environment and social issues

ESG is an issue that is challenged with the balance of social goals while maximizing financial value creation.  This podcast is a great introduction to practical ways to comply with ESG demands.

In the commercial world companies estimate annual sales and make a budget. Hard to apply basic management 101 to the Continuous Resolution situation that many federal agencies have found themselves in the past few years.

Today’s interview brings together several experienced federal professionals who give guidance on the new world of budgeting. Issues brought up include critical timing concerns, technical definitions, and the impact of shortened periods to accomplish annual goals.

Timing. An agency can set a budget, then not know when, or how much, they will get funded. Somehow, agencies manage to get by, but most of the lessons learned are part of institutional knowledge of each respective agency and are not shared.

For example, what if an agency plans an expenditure, then only gets 75% of what was planned? If a chunk of a budget is eliminated, how does an administrator prioritize what projects to continue and which ones to cut?

Definition. Under a CR, an agency cannot begin any projects. This leads to the legal parsing of the meaning of “new.” If a system replaces an existing system, is it new? When a system is maintained, is this a new project? If a vulnerability is found, can it be remedied with a new patch?

Compressed year. How long will the CR last? 30, 60, 90 days? Does this mean that each agency must produce an administrative plan for each shortened year?

The interview ends on a bright note. Elizabeth Field shows the GAO understands the challenges this presents. They have issued a report called Selected Agencies and Programs Used Strategies to Manage Constrains of Continuing Resolutions.

An argument can be made that the Solar Winds breach precipitated the interest in Zero Trust in the federal government.  Thousands of words have been written, justly, about the incident.  However, much less attention has been given to how SolarWinds has handled the situation. 

SolarWinds has provided us with a classic case study on how to handle a crisis.  They have been transparent, changed leadership, and have made strategic acquisitions that help them serve customers better. 

An example of that strategy is today’s interview with Gregory Fetterhoff, the CEO of Monalytic. It was acquired by SolarWinds and operates as a separate company. An argument can be made that new leadership at SolarWinds has objectively looked at how to improve service to federal customers and made the acquisition of Monalytic to remedy the situation.

During this interview, Gregory Fetterolf gives three reasons why this partnership is effective: the skill set Monalytic brings, accommodations made for corporate culture, and the synergy gives Solar winds the ability to serve the federal government in other areas.

Monalytic is comprised primarily of people who have served in the military or federal government.  SolarWinds had extensive experience in commercial environments.  The credentials that Monalytic brings to the table allow them to have a deep understanding of federal needs.

Consultants like to say culture eats strategy for breakfast.  What happens when two completely different cultures get thrown into the same room?  Leadership at SolarWinds has the confidence to allow the successful culture at Monalytic to continue, garnering respect from all employees.

The strength of this new partnership is revealed in the interest of both federal as well as commercial organizations.  From the government side, they appreciate the ability of Monalytic to understand their needs.  Commercial entities know that federal compliance is difficult; if they start with federal compliance, the commercial concerns go away.

Sometimes, it takes an event to show the true strength of a company.

The three subject matter experts in this discussion give the listener a wonderful perspective on challenges and solutions to moving to Zero Trust.

The interview revolves around tools needed to audit a network, risks inherent in a hybrid cloud, a why a Zero trust platform gives an agency the flexibility it needs to deploy zero trust effectively.

Every discussion about zero trust for government agencies starts with trying to determine what is on your network. Smurti Shah from Michigan notes that tools that commercial organizations can use to accomplish that task may not work in a government environment. Therefore, State and local organizations must select Governance, Risk, and Compliance (GRC) solutions that are permitted.

Ian Farquhar from Gigamon brings up a fascinating issue with the “discovery” aspect of network analysis: cognitive bias. For example, a systems administrator may swear on a stack of bibles that they have documented every single item on the network. Ian mentions simple questions like: What about that copier? Does it ever have sensitive documents on it? What about the printer? If your organization allows employees to bring in devices, what kind of security implications does that bring?

During the discussion, the concept of “trust” was unpacked. We know that trust applies to “who” and “what,” but what about the system itself? Ian Farquhar applies trust to logging and Cloud Service Providers (CSPs).

The Solar Winds event looks like it started with the modification of the logs themselves. If you trust the logs, then you can be vulnerable to attack, one should apply zero trust to log controls.

One approach to minimizing vendor lock-in is to use a hybrid cloud. This adds complexity to an already complicated situation. The CSPs certainly do a wonderful job at telling people about the security of their cloud. Be careful to apply controls to that cloud environment, offloading trust to them can put you at risk.

All participants agreed that zero trust gives the flexibility to handle attacks today and in the future.

When the World Wide Web was developed in the late 1980s the idea was you could get information from the platform easily. Well, that certainly worked. In the past thirty years, this “interconnectedness” has brought benefits and unexpected risks.

One of the dangers is the ease of one system connecting to another. Great if you want to validate a person’s identity; also beneficial for a malicious actor to place code in a system that automatically updates.

The most egregious example of the danger of automatic interconnectedness is the Solar Winds event. Systems were set where network “A” trusted code from network “B.”  The cyber attackers took advantage of this trust and inserted code into the target system.

The application for systems managers is obvious – if your architecture is designed to connect to trusted third-party solutions providers, how can know the code is clean? If you combine that with the lack of staff that most state and local governments have, then you have a serious problem. Just to amplify the situation, remote connections blossomed in COVID and the number of logs to manage is out of control.

This is a discussion where subject matter experts from software companies, federal leaders, and county practitioners sit down to provide some suggestions to solve the vexing problem. One possibility is to treat code in a “suspected” manner. Take each system update and consider it as malicious and run it in a sandbox before deploying. Unfortunately, this is a labor-intensive process, and we are assuming a situation with a lack of professionals.

Bill Harrod from Ivanti suggests that systems administrators can take advantage of artificial intelligence and automation to vet patches and updates quickly. If there is an issue, remediation can take place rapidly.

Another remedy discussed was including text in future contracts where software vendors must assure end users that the code they provide has been thoroughly evaluated. This does nothing for a system in place today, but it is a good long-term preventative measure.

This interview is a terrific primer for preparing your agency to prevent ransomware.  We have four experienced thought leaders who discuss issues like knowing what is on your system, where to find free prevention resources for a limited staff, and best practices for maintaining a safe network. 

Knowing what is on your system is the basic starting point for security, however, this simple concept can be difficult to accomplish.  Auditing your system can be compromised through users avoiding consolidation through shadow IT.

The term “shadow IT” has had Its peaks and valleys of interest in the past decades – some refer to it as “unauthorized modifications.” The origins are obvious.  If a system administrator makes it onerous to comply with security directives, users will come up with a workaround and use a credit card for an application, unknown to management.

The federal government understands the staffing challenges of state and local governments. As a result, the Cybersecurity & Infrastructure Security Agency (CISA) provides guides and training for these groups. During the interview, we learn that CISA is developing guidelines for an attestation letter.  This would act as a trust mechanism for smaller agencies concerned about malicious code coming from vendors.

Even if your system is thoroughly reviewed and current with updates and patches, you can still be vulnerable.  One simple instance of best practices is offered by Bill Harrod from Ivanti.  He suggests that each update must be tested and validated before being installed.  This is because of the complicated nature of hybrid systems today.  One update can have an impact on another system that is dependent on it.

Much to unpack in the group discussion – they talk about automation, machine learning, and the software lifecycle as it is applied to the software supply chain.