Plan for the worst: preparing for security incidents

This article is all about planning for security incidents so that their impact is minimal. Small businesses often rely on common materials for both security and operational incidents, so hopefully the information here can be utilized on the disaster recovery front as well.

Preparing for incidents can be time-consuming and challenging, since you need to organize a wide range of people to participate in the planning, documentation, and testing of the incident response plans in order to derive value from them.

However, it’s important to consider the customer’s perspective. They are paying for a service that may be critical to the success of their product. Downtime and incidents will upset customers, and slow response or a lack of professionalism during incidents will likely drive them away.

Spending a few hours every now and then to walk through how major incidents would impact your product can open up discussions about how to best serve customers, help to discover previously unknown single points of failure, and, most importantly, give whoever is first to handle the incident a few hints to get the ball rolling as quickly as possible.

Keeping a record of what happened is critical to understanding the root cause of any incident. This section is first because troubleshooting an incident or identifying a root cause may be impossible without those logs. Audit logs in a security context are there to help your business keep a historical reference of key events, such as authentication, changes to management settings, or modification of data. A common example is the audit and investigation logs produced by Google Workspace, which include tracking events for admins making system-wide changes, updates to group membership, or when a user has authenticated. These audit logs will often contain critical data such as the user who performed the action, what happened, the timestamp, what asset the event occurred on, and the IP address.

Basics to get started?

Make sure that all the important systems within your business have audit logs enabled. Most will have these enabled by default, but it’s important to check that they’re available to you and how far back you can view the events. For example, Google Workspace will only keep admin log events for 6 months with no ability to make it longer or shorter.

Where possible, set up alerts that can be generated when something bad happens. Using Google Workspace as an example, it generates lots of alerts based on a predefined list of known threat patterns, such as a suspicious login or a user being granted admin privileges. Set up these alerts to send them somewhere that you will monitor, like a Slack channel or an email distribution group.

Whats next?

Centralizing logs in a single location will be important as your business begins to onboard more systems and your compliance requirements change. This will save time when triaging a single incident that may span across multiple systems, and it will also allow you to create alerts from a single location rather than across multiple systems. When looking at PCI-DSS, you must keep logs that could impact credit card data for a minimum of 12 months. As we saw earlier with Google Workspace, this may require exporting the logs to another location with longer retention.

What does Kinde do?

All of Kinde’s infrastructure logs use AWS CloudTrail and CloudWatch, which are replicated to a separate audit account that is more restricted. This is to ensure that nobody accidentally deletes the logs while performing maintenance and reduces the risk that attackers can cover their tracks. Our sprawl of systems that handle sensitive data or interact with sensitive flows is limited, so we haven’t had a need to look into centralized log storage yet. All alerts from any system, such as CloudWatch or Google Workspace, are sent to a few Slack channels depending on the context.

Dealing with a security incident is unpleasant. At the worst end of the spectrum, someone may be inside your house and rummaging through your belongings. You need to get them out as quickly as possible, prevent them from re-entering, and take inventory of what they have stolen. There are many resources available to help you handle security incidents in detail from large industry groups, such as SAN 504 or NIST 800-61. However, I have found that it is much easier to get participation from other people in the business when starting with something smaller, such as this incident response plan for startups guide.

Basics to get started?

Start with this boilerplate incident response plan. It has a simplified framework to get you started. From there, customize the plan for your own business and your customers’ expectations. We started by creating a Slack channel for communicating incidents, a very rough plan based on the template, and adding language to our awareness training that lets everyone know in a few sentences who to contact if there’s a security alert. Keep it simple.

Once the template is complete, schedule a security incident tabletop with individuals who may be responsible for the infrastructure, identity, IT systems, and product delivery. It will be a delicate balancing act to ensure the right mix of disciplines without having a massive meeting. Challenge the group with a few possible incidents, such as someone outside the company joining the Slack workspace, Google alerting of suspicious logins, or malware detection on the application servers. Discuss what you would do as a team and refine the response plan and playbooks.

What’s next?

Run a simulation of a security incident. There are some fun tools available that can help, but it could be as simple as creating a test account and performing a safe change somewhere important. This challenge would test how everyone would respond and check the basics, such as whether there’s an alert to cover the event or if there’s even a list of people to contact during incidents.

What does Kinde do?

We used the incident response plan for startups as the template for our own security and privacy incident response plan. So far, we’ve conducted two tabletops exploring scenarios like malware, phishing, dodgy code commits, and identity theft for our admins. Our plan is to conduct two or three of these each year and focus on risks that may impact our business. All the tabletop results feed into the creation of playbooks, fine-tuning of our response plan, and identifying risks that we weren’t addressing properly.